Thanks. Looking forward to trying it out!
@general background (for other readers) :
(My codex making skills are improving, but the errors I encounter due to sloppy OpenAI, or personal misunderstandings might be useful to others. )
The message Jim replied to wa s:
… I (Alison) have been dry-running the codex inside subtxt. It worked reasonably well, but there were a few persistent error type s.
1) timeline: if I want to do prequels and sequels, I need before and after snapshots of the world. As of now, Narrova keeps reverting to the current story world. Makes sense, but not what is needed. So, I need to update the codex for these different conditions. Easy enough, but a bit of work which I need to do.
2) Whatever has happened in the stories that I used to generate the codex keeps on happening again in new stories generated. Again, makes sense, but not what is needed. I need to separate established story happenings from locations and places to stop them from repeating in newly generat ed sto ries.
3) Keeping characters consistent with their established personality: It’s a tight balance between an established personality and a new dramatic function into which they might be placed. In my dry run the character Shams, f.ex., totally changes his established personality to fit the dramatica needs. To avoid this I need to recreate the character codices to firmly state their personal ities.
4) Important: Keeping characters role neutral in their codex descriptions. I based the codex on already written story, which has a protagonist MC (Risha) and a few logical influence characters. In the dry run, I purposely defaulted to different MC and IC characters, but Narrova kept pushing me back to the already established book MC and IC. The new story and the story upon which the codices were based thus conflict and cause troubling errors and contradictions. I need to recreate the character codices to be role neutral. (Now solved b y Jim!)
(end of message)
A few things I learnt that might hel p others:
- Claude is more versatile and delivers far better RAGs than OpenAI. Way better. I can load loads of files in one project, which it then handles intelligently and coherently. However, Claude easily reaches it’s max usage under a normal paid subscription, which causes lots of downtime.
- OpenAI: really needs the (extended) thinking turned on for codex generation. Can only handle 20 files. Best for distilling the gist of things. Claude tends to stay more literal. OpenAI really needs to be told to stick to the source material and not add its own crumby inter pretations.
- NotebookLM loves short responses, even when I ask for complete and thorough. (I use the free version). So, I gave up on th at for this.
- The message reprinted above described the end of my first round of RAG-generation. I have my books well-tagged in Causality software and output each tag to a docx, which I then supply to an AI. I have now learnt that this first run is necessary, but preliminary.
- My second round, I am doing in Claude. I uploaded all first generation RAGs to a project and in the instructions I mentioned the above 4 error types. I also asked to check for overlaps. It understands beautifully: removes the redundancies and solves the above error types. I have it generate each RAG-type individually in the same chat. Costs me a lot of tokens/downtime, but keeps each RAG-category as a whole system.
Work in progress. Corrections by those who know more ar e welcome