The PolyBlog

I wrote earlier this week (Using Calibre to embrace my inner librarian for ebooks) about the Poly Library 3.0, and when I did, I thought I had most of my “work” done. I had decided on three main areas (the book profile, user engagement, and user tools), although, truth be told, I had four categories that were more easily explained as three…I feel like some of the user engagement and user tools could theoretically be separated into a fourth category, but I digress. I had also decided on about 35 new basic fields, though that number will grow once I start adding visual icon fields, etc.

But I asked some other questions on the Calibre sub-Reddit, and the answers sent me scurrying into lots of different additional areas. Squirrel mode activated! Albeit in (mostly) a good way. 🙂

In the meantime, I dropped about 1500 books off at Value Village today. My paper library is almost decimated. I still have about 400 or so, but the rest? Gone. A project I’ve been wanting to do since 1998. Just finally had the chance to do it properly over the last few weeks with more time at home with Jacob. Now it’s on cleaning up my ebook library. Oooh, and a friend dropped by last night with his daughter and took about 50 books away with them. Not counting the one I gave to a friend across town last month, and about 10 that went to Jacob and Andrea’s library. I would love to have had time to find new homes for all the individual books, but hopefully readers will find them at VV.

Some basic structural things to work out

One of the first things I need to look at is “nested hierarchies”. For example, if I used FICTION and NON-FICTION as level 1 tags, I could then have a subset of tags under FICTION for the different categories. Similarly, another set could sit under NON-FICTION. The ideal part of that is it makes things really easy to do subsets together. All fiction? Easy. Biography only under Non-Fiction? Easy. Historical fiction AND biography? Two clicks instead of one. This feeds into a larger problem I’ll discuss at the end, though.

Secondly, I need to figure out what I’m doing for Icons for various tags — rather than a field that simply shows FICTION, I’ll likely add an icon that shows Fiction vs. Non-Fiction…maybe a magic wand for fiction and a # sign for non-fiction or something. I have lots of choices, and the actual icon choices can come later, but for now, I need to start thinking WHICH fields will also have a second field with an icon to represent that category. That way I can hide the column in a larger set and JUST show the icon instead. Even for something like # of words, I’m tempted to use a series of icons for thicker and thicker books depending on a range of sizes. Oddly, enough, as some of these are formulas to do different things, I also have some other formulas I want to include. For example, some of the basic metadata uses “dates” for things where I don’t need the actual date with day and month, just a year. Do I care what day of the year a book was added to the database? Nope. Not usually. Do I care what day of the year a book was published. Almost never, and it isn’t often accurate. The day of the year I wrote a review? Probably not the DAY, but yes, probably the month and year. Maybe even similarly for when I read stuff, although that might be more about current reads than old reads. I have no idea when about 300 books were read more than just approximate year, but they’re all in my pending review folder. Or at least they used to be before I borked everything. 🙂 Hence the opportunity.

Third, I probably need to make a hard couple of decisions about how I’m integrating my Library output into my WordPress site. Right now, I have 7 custom fields that sit at the bottom of all my reviews on my website. They’re hidden, you can’t see them, but they generate all the links on my other pages to see books by publication, by BR #, genre, author, etc. Most of that is also directly recorded in Calibre, and to some extent, an even larger consideration with OneNote so I don’t lose text. But…what if…instead…hmm. Yeah, I *could* put all that data into a slightly different format in Calibre, add TablePress into the Website for all completed reviews, and bob’s your uncle, I could generate a full data dump (about 300 books worth of metadata) in a limited form into a CSV format and paste into the TablePress plugin, which would then update all the data across the site. It wouldn’t solve all of my integration needs, maybe a third. It would, however, stop me from having to code any page with extra metadata to generate the links. I’d lose a bit of functionality, but the TablePress tables DO allow for easy filtering and searching. Hmmm…

Another third of that integration question is whether I do anything with my reviews. Currently, my reviews are built on (mostly) four big sections — plot/premise, what I liked, what I didn’t like, and a one-line review. There’s a fifth piece for some around disclosure, and then we also have elements around the rating, etc. If I include the coding above, call it 11-13 fields or so. If I’m going to redo this from the ground up, why not build the review format I want directly INTO Calibre and add custom views that would show me the whole review in HTML? Ready for pasting into the website or elsewhere?

The last 1/3 of that little integration puzzle is if there is anything I should be considering around “up next” or “currently reading” or even just a list of all the authors I have in my larger database (the list is huge). I don’t know if I want it “public” per se, but I do like the idea that somewhere online I have a simple list of all the books in my database. Some people run it as a server and can see all their books online anytime they want. But I don’t want the actual books, not really. I just want a version of the larger list. Sure, in theory, I’d love my entire database online, but then it is tempting to start sharing, opening it up to friends, encouraging piracy, etc. Nope, my books. FOR BOOK HALLA! (the cry of a Book Goblin)

I may need to re-learn how to read

Okay, that’s a small joke, as what I really mean is that I need to think a little bit about the process of getting books from my computer to the readers (I have two main ones) and back again.

Here’s the thing. I have a lot of ebooks in different formats. Many of my older non-fiction books came from various sources, often in PDF format. I could try converting them to epub for better reading on my Kindle, but sometimes they have diagrams that would look way better on a larger tablet. Which I now have, after repurposing a Galaxy Tab S2 with a 9.7″ screen (separate posts incoming!). Except that I also want to annotate some of my reading as I go. You know, highlighting and stuff? I have an easy way to do that on the tablet. BUT then how do you get those comments back into Calibre and saved without having to re-add the book? Oh, right. An option that may link into my third big area. Another element to think about.

For Kindle, it is relatively okay. When I sync with Calibre and then potentially run a plugin called annotations, it will look to see if any of the books on the Kindle have annotations/highlights/notes/etc, and import them into Calibre. Needs some tweaking and streamlining for setup, and I might have to do some things in a specific way, as I read, but it works.

Yet again, though, there is an element of WHICH books go on the Kindle that relate to the third element that I have to work out in the last section.

I found some tips and tricks online that were really interesting, and something I never would have thought of on my own. Let’s say I create a field that has Private Detective, Amateur and Police as three types of mystery stories. The navigation sidebar will let me have an option where I can click on those values and see all the private detective books, amateur detective books, and police detective books separately. A filter, if you will. Except until you have a book in the library that USES each of those categories, you only see the options that are already populated. If I only have one book that is tagged private detective, and no amateur or police books, those two headings don’t show up at all. It’s not only “0”, it just doesn’t show as an option. So someone came up with a fabulous trick. They create a dummy book they call DO NOT DELETE — DUMMY BOOK and they include ALL the possible tags in it. Which means that every category will contain at least one book. The dummy one. This is INCREDIBLY useful when doing initial intake, and I wish I knew it YEARS ago.

The same user described another workflow issue that I had never given much thought to, to be honest. Let’s say the final profile of the book has maybe 70 fields. I don’t normally populate ANY of the extra fields until I’m done and going to do the review. By contrast, a lot of people tidy up all the metadata before they add it to their main library, which makes perfect sense. One challenge with downloading data later is that tag fields are filled with everyone else’s tags, whether accurate or not, and added to your main library; if you clean it first, your main library remains more uniform.

Oddly enough, I also loved one of the user’s metaphors for their workflow. They called their “intake” area “DECON” where they cleaned up all the data. Then, when they moved it to their main library, they call that Alexandria. That’s quite cute in my view. Not sure what I’m going to use, but I’ll think of something. Even if I use virtual libraries and put the metaphorical titles there.

Another user has customized their “intake/decon” process so that any book added to their library not only gets all the fields, but forces a number of them into default levels. I just left them blank, without thinking too much about it. Even “Intake” was often me taking a whole bunch of books that had NO TAG at all for workflow and moving them to INTAKE. But I could just say, “Hey, any book added that doesn’t have a WORKFLOW tag automatically gets the WORKFLOW tag set to INTAKE. Would have saved some steps in many cases.

I’m intrigued by another user who has a library of books they still want to GET. No files, just the name and author and why they want it or where they heard about it, some sort of note field. I don’t see the advantage of that over a simple note list, other than sorting. You’d end up doing a lot of metadata for a record that will likely later disappear unless you merge it with the file, I suppose. I don’t know, it sounds redundant to me, so I asked them for more details on how they use it. I like the idea of a list of books that I don’t have yet, particularly for series.

As an aside, reading through the Calibre subReddit is fascinating to see how people create their own workflows and metadata, plus icons and colour coding (I don’t know how to colour code columns yet). I don’t yet know if I will use any of them, but here are some examples:

• People with a “read” status that I would think was simply “To be read” and “Read”…nope, they’ve added Unread, Read, Read enough, Try again, Do Not Finish;
• For variations on that one, they often add a second tag with status like To Read, Up Next, On Hold, Reading, Finished, Abandoned, Reference (I use a few of those);
• Another user created a “vibe” category for their “Next” books to read… sounds fine, but then they listed all the steps they take, which weren’t minimalist, and then said, “They like to keep it simple!!”;
• A surprising number of readers have added columns for the number of times they have read a book…Jacob would benefit from this dramatically, having read several of his series multiple times, even the huge ones, but I am out of time for age — I am not going to reread anything new again…I might revisit some old books I read, but I doubt anything new will get re-read before I die!;

I thought I was done playing with my metadata field choices, but well, you are never “done” in librarianship, right?

Deep breath, talk about the elephant in the library

Sooooo, there’s a small basic question that I haven’t answered yet. How many libraries will I have in Calibre?

For those who don’t understand Calibre or ebook software, full libraries are kind of like having different rooms. You might keep all your biographies, for example, in your study next to your reading chair and fireplace. Things you read more slowly on a cold winter’s night. And then, perhaps, you have contemporary stuff in the family room, more light-hearted fare that you pick up and down at will.

For me, the big division starts with the simple distinction between Fiction and Non-Fiction. But it quickly devolves into other questions. In my previous library, a single room to hold all the books, I had the equivalent of separate bookcases in the room that were divided by workflow. Not unlike a real library. There was shipping/receiving, where the books arrived and were placed on a shelf (called Intake). Then, I would put them in a general sort between Fiction and Non-fiction as I read those in very different ways, and at different frequencies. But then, as I started to “process” them to add to the library, I would put them into sub-categories so all the mysteries were together, perhaps with standalone books sitting differently from books in a series. Almost like moving them to other bookcases. Followed by active bookcases when I actually started to put them in my TBR pile on my Kindle. Sorted even more granularly on my Kindle, with subcategories for Mystery, Fantasy, Non-fiction, Contemporary, Other, etc. Plus a folder for READING RIGHT NOW (not actually called that, but basically I have 300 books on the Kindle, with no real order to follow in advance other than what strikes my fancy when I finish one and start another, but they can’t ALL be in the same folder, that’s just crazy talk). And then when I was done, I had a separate workflow for NOT YET REVIEWED and another two for FINISHED – FICTION and FINISHED – NON-FICTION. Plus others for reference or DNF (did not finish), although often as not, I just delete those.

Here’s where the rumble starts. Some Calibre users are very much of the Texas Rangers motto, “One riot, one Ranger,” and have a “One reader, one library” approach. Others are more into separating things into libraries by likely either workflow or subject matter…more of a “A place for every book and each book in the right place” approach. And then there are the alternately simplistic or sophisticated users who go with a hybrid approach called “virtual libraries”.

I say simplistic, as many who like the virtual libraries model also like to have hundreds of possible tags to sort things in metadata anyway they can. So, for example, if they read To Kill A Mockingbird, they would likely tag multiple sub-categories, with something like “American, literature, classic, law, lawyer, coming-of-age, fiction, racism, history, trial, YA” and then play with various virtual libraries for some of those, like an applied filter to an open-ended keyword search. So, for example, a virtual library containing all the law-related books. For many, they see it as the best of all worlds — a giant library with a way to only “see” the books in certain preset categories. Sounds great. But they often find after a bit of use that they have one library but are starting to use some of the virtuals almost like workflows…there’s almost no benefit to the “virtual” side over separating into distinct, more manageable, smaller libraries. If I use the Kindle as the example, my “active list” of books on my Kindle exceeds 300. That’s ridiculous. I’m not going to read 300 this year, not even the next five years, so wouldn’t it make more sense to prioritize that into smaller libraries of what is ACTUALLY active and likely to be read this month, even if only to improve my Kindle management? Or my tablet for non-fiction.

I also say sophisticated as some have come up with really good reasons for using virtual libraries, not the least of which is a library for a specific reader. If I take a book like Anne of Green Gables, that one’s relatively easy. I don’t have a big interest in it, nor Jacob, so if I wanted to put it in a separate library for “Andrea’s books”, that would make sense. Alternatively, I might have Harry Potter, which all three of us have read. So, would I put that in a “shared library” or put it in each of our libraries, duplicating it three times? Or one copy as if it was MINE, and the other two would only “borrow” it (more about ownership).

For me, I am strongly attached to the separate library model. I love the workflow aspects of it. But then I run into a problem almost immediately. Let’s say I’m reading Ben Aaronovitch’s Rivers of London series, recommended by our friend Paul (that’s not an euphemism for me; it really is a friend called Paul!). I’ve read and reviewed three of them so far. Three more are still in the “to be reviewed” stage. Another four are, I think, in the TBR pile. So if I want to see the whole series, they would be in potentially at least three separate libraries. If I see a book on sale, and I want to see if I have it or “need” it towards a series, I can’t easily do a search of all the libraries. The virtual library lets you do a SUBSET of your main library, not combine multiple libraries. To me, that would be the ideal — separate libraries and then one ring to rule them all and in the darkness bind them. Alas, that doesn’t work.

As a result of many of the little elements already mentioned above, I need to decide almost at the beginning which way I am going to go — separate, main, or main with virtuals. For example, if I went with intake, that would need most of the initial fields for the book profile, but almost NONE of the other fields in the full set. If I then separate into FICTION or NON-FICTION libraries, then fiction doesn’t need the NF categories and NF doesn’t need the Fiction categories. If I eventually have a “reviewing” library, I only need to start adding the review fields at that stage. The final library probably has all of the fields, although some of the process stages might even disappear then, too.

And it creates a dilemma for me. The fact that I couldn’t search across all libraries at once was enough of a pain that I would occasionally search my TBR for a new book I saw, not see it, and buy it … again. Because I had already downloaded it and stored it in another library. By contrast, having everything in one library is how I borked the current tagging. Separate libraries would have prevented that specific issue BUT I could still bork it other ways, just as easily.

Decisions, decisions. And honestly, using virtual libraries doesn’t REALLY help that much. My workflow tags were the equivalent of a virtual library anyway, as I forced SINGLE options into that field. I wouldn’t let the book be tagged as both INTAKE and a TBR category, for instance. Clicking on one sub heading essentially gave me an instant virtual library anyway. Actual virtual libraries are usually designed to be MORE complicated than that, but also allows you to use multi-book commands on the sub-library without combining them with a search. Just click on the virtual, it’ll show everything for intake, and bob’s your uncle. You can even set custom views, so that all the other fields will be hidden. However, you CANNOT have separate field lists for the book itself — if the total number of fields is 145 across all the various workflows, it will have all 145 in all of the books. This increases the size of your database, but not problematically in this day and age of cheap storage.

I’ll have to figure this out pretty soon. Interestingly, there are a bunch of people who suggested not to decide. Just play with it, merge or separate later. Except I do have a big problem up front. It’s the process I mentioned for reading non-fiction books. If all the books are in one library, then that whole library generally has to be located in the exact same place. Think of it as a master root folder for all the subbooks organized by author and then by books. A file structure on the PC drive, if you will. Except that creates a problem for annotations on NF books on my tablet. If I DL the books to the tablet, read them, annotate them, and then want to save the annotations, most of the solutions involve re-uploading that annotated file BACK into the library. Not the cleanest of solutions. However, if non-fiction is a SEPARATE library, AND I choose to save the library in cloud storage like OneDrive, then any changes I make to the file will directly go to the saved file in Calibre. It’s the same file. All annotations automatically in. All saved. Great, right? Except then I can’t have a merged library AND I don’t actually keep the original without annotations.

Somewhere in there, my brain just exploded. Maybe it’s because it is too late at night.

Regardless? Deciding on virtual vs. physical libraries is job one.