Searching Daily Kos
Quick Search Tips
Search: enter words you want to find in the target collection. By default only targets containing all the words will be returned. Capitalization is ignored. If you want to find several words in a precise order (a phrase) put double quotes around them as in "this is a phrase". Single quotes are not permitted, and are removed. Although "AND" is assumed by default, "OR" and "NOT" are supported, as follows.
Multiple query words are by default understood to be joined by "AND". Certain characters have to be "escaped" by placing a back slash in front of them; to find "/", use "\/", to find "=", use "\=". Complex queries can be constructed using parentheses and restricted queries. Restricted queries: limiting the parts of the document to be searched. The available options are described here. For both stories and comments (documents):
For stories only:
For comments only:
Find: In order to choose what kind of documents to find, use the pull down menu to select one of the following. Each option is identified as either a database search, meaning that the primary database is searched using Scoop directly, or index search, meaning that a specially constructed index of words contained in documents is searched using swish-e. There are three different indices, one for diaries (aka user diaries), one for stories (aka front page stories), and one for comments. User diaries that are elevated to the front page are found in both story and diary indices.
From: and To: pull down menus restrict the time interval of the search. The default interval is from one day ago to most recent. Most recent may be as much as 5 minutes out of date. Sort by: The order of presentation of results. Not all choices are relevant to all searches. Story and Diary Search sort choices.
Comment search sort choices shared with stories and diaries are "Relevance", "Title", and "Time", which are as described above. Those specific to comment are "Cid", "Nrec", and "Ntroll".
More example queries
For comments only Everything past the "#" is a comment, not part of the query!
Searching Daily Kos with election tags (etags)You can now search using election tags as follows. etag=CT-SEN Etags with embedded spaces should be quoted. etag="CA-LT. GOV" You can only use one etag per search. Good election tags have a two letter postal state abbreviation hyphen seperated from either a house district number, or a statewide office abbreviation (SEN, GOV, Lt.GOV, SOS, etc). There is one exception, etag="2006 ELECTION" also works. Good etags are ones we already know about by finding them in the tag list of an existing diary or by appearing in one of the election roundup diaries. The etag is used to build a more complicated query that looks for the tag in tags, title, or body text (stories/diaries) or in subject and comment text (comments). Certain meta dairies are excluded, including open threads, jotter's daily lists, and sidinny's election roundup, and the top comments diaries.
Software used.The search facility currently installed is based on the swish-e search engine: it is fast (compiled C code), open source, and flexible (you can feed it documents out of a database using your own custom preproccessor). It is intended for document collections numbering a million or less, which will do for Daily Kos with under half a million documents (excluding comments). In addition, swish-e offers many of features most desired, including search of phrases and search within configurable document areas (title, author, tags, URLs). What is indexed. All Daily Kos documents from the beginning to within 5 minutes of the present moment are indexed. Older documents are stored in static yearly indices while more recent documents are indexed and reindexed, frequently at first, then less frequently as they age, until they roll over into a growing, static, permenant yearly index. The right mix of number of index files and frequency of update to obtain the optimal combination of ease of maintenance and speed of search may require further experimentation. At present three different series of indices are maintained, corresponding to (in order of increasing total number of documents) front page stories, user diaries, and comments. For 2005 the number of stories, diaries, and comments was 5097, 88943, and 693224, respectively, containing 56993, 371578, and 2945916 distinct words. Design Considerations. The utility of search to a community of users is delivered not simply by the results of the text search, but most critically by the ordering of the results such that highest "value" hits are displayed first. (Most search researchers know that no one looks beyond about the first handful of results; almost no one looks beyond the first page.) The key to arranging for best to come first is to understand, have access to, and use the meta data associated with the text documents searched. Meta Data. At Daily Kos, we have a number of sorts of meta data that are relevant to documents. First among these is the time of posting. What is the latest? When did they say that? Who was first? That sort of thing. So the one thing search has to get right is time, in the sense of being up to date, that is including the latest diaries, as well as in the sense of permitting searches to be organized by reference to the present time. Making it up to date. Because recommendations are only open for 24 hours after a diary is published, it is important that search indices be refreshed sufficiently often during that interval to provide reasonably up to date information on total recommendations and comments. Currently, diaries up to 6 hours old are refreshed every 5 minutes, and diaries 6 to 24 hours old are refreshed every 15 minutes. More Meta Data. Other important meta data associated with stories and diaries includes comments and recommendations. The number of recommendations offers insight, imperfect, but useful, into the degree of approval a diary has received from the Daily Kos community. The number of comments indicates the size of the discussion engendered. The two numbers together give an interesting insight into a diaries history at Daily Kos. Some user diaries are promoted to the front page, and some diarists ("front pagers") are allowed to post directly to the front page. The latter are not open for recommendation. In order to harmonize listings from searches which include both front page stories and user diaries, a plausible value for number of recommendations (83) is assigned to front page stories, but in lists, this number is indicated by "*" to avoid placing too much emphasis on this made up number. The search query. Search queries can contain individual words, all of which must be found in a text document to constitute a hit. Queries can also contain phrases, indicated by double quotes around the phrase. All words are converted to lower case while searching, so letter case is of no consequence. In addition, queries can include boolean terms, such as "and", "or", and "not" which can be used to construct more complex and perhaps more informative results. By default "and" is assumed between individual words in a query. Finally, queries or parts of queries can be restricted to only search within parts of a document or document metadata. The parts available are shown above, along with examples. Certain characters have special meaning to the search engine (open and close paren, double quote, equals sign). Only a restricted list of characters are expected to occur in a word. For the indices here at daily kos, the word characters are as follows.
All other characters (non word, non special) are silently replaced by spaces while searching. For instance searching with "Jerome a Paris", "Jerome.a.Paris", or "Jerome,a,Paris" all give the same result. This is important in certain cases, for example searching for the url http://riverbendblogspot.com is accomplished with the query site=("riverbend.blogspot.com"). The reasonable looking but incorrect query site=riverbendblog.blogspot.com doesn't work as expected because in the absence of the double quotes and parentheses, replacement of "." with " " and the default boolean AND understood between search words makes it equivalent to site=riverbendblog AND blogspot AND com, which returns as hits only documents with riverbendblog in a url, and blogspot and com in the body text. More on Search
If you are intriqued by search and would
like to know more about the basics of how it is done, this is a
good introduction.
On Search. Basic basics. by Tim Bray. Since I've emphasized the importance of meta data, I'll also point to another article from the same series, On Search. SearchMeta, with some further thoughts on things meta. While I'm on the subject of underlying technologies, let me end this section by pointing to an article by Tim Bray on Ajax, the other new addition here at Daily Kos. In it, he also gives credit to Jesse James Garrett for the Ajax acronym in the process of thoroughly describing it.
Daily Kos is an amazing place, community, phenomenon. It has been my
privilege to have the opportunity to contribute.
-- jotter |
Recommended
- America’s mayor posted this. It has since been deleted… But the Arizona AG was on the ball. Because less than 30 minutes later….14 comments 53 Recs
- My, my, my, the Manhattan Criminal Courthouse has become the place to see and be seen for the proto-fascist sycophant set. It’s like Studio 54, only for excruciatingly dorky, power-hungry nitwits. “Hey, wanna play hooky from Congress to help an...39 comments 180 Recs
- Reposted from Dark Skies on the Horizon It’s my opinion, subject to confirmation, that Donald Trump will lose the election interference case. He will be convicted. The defense is attempting to ...190 comments 367 Recs
- Welp, Donald Trump took the bait. He once again let his untethered ego take him where he doesn’t want to be. Into a pair of debates with a very smart president……..on CNN! And ABC!!! Donnie always wants to be the aggressor. Problem for Trump is, Joe...193 comments 545 Recs
- First the grey elephant in the room: The NYT not only released a ridiculous poll this week, but then they milked it with follow up stories on why Biden is losing so badly and why everyone is ...18 comments 28 Recs
- Ukraine launched a broad attack with drones and missiles against Russian facilities in Novorossiysk, Tuapse and Sevastopol. The oil refinery in Tuapse, which is about 450 km down the Black Sea coast from the front, was set ablaze just a week after it...90 comments 396 Recs
- The Supreme Court Justice who wrote the opinion overturning Roe v. Wade displayed an upside-down “Stop the Steal”-intentioned flag at his house, less than two weeks after the Jan. 6 insurrection, purportedly in order to protest anti-Trump signs a...337 comments 691 Recs
- Whatever one’s views of the acerbic Democratic strategist Jim Carville, his 1992 “It’s the economy, stupid” has become a cliché. That sentiment apparently resonates with voters, including the young ones not even born when Carville said it, as this...117 comments 338 Recs
- Late Night Snark: Verbal Pickleball Edition “Donald Trump lost two debates to me in 2020, and since then he hasn’t shown up for a debate. Now he’s acting like he wants to debate me again. Well, make my day, pal. I’ll even do it twice. So let’s pick the...153 comments 163 Recs
- Russian forces conducted a limited series of drone strikes against Ukraine on the night of May 16 to 17 and individual missile strikes on May 17. Ukrainian Air Force Commander Lieutenant General Mykola Oleshchuk reported that Russian forces launched 20...11 comments 79 Recs
- This is the 20th anniversary of Freewayblogger.com — a site I haven’t updated in several years, but whose headline photo and opening emphasis on the Russo-Republican Alliance remains as relevant today as ever. As does the philosophy behind the format:...28 comments 210 Recs
- Marjorie Taylor Greene has, arguably, descended into new lows for uncivil behavior. And that’s saying something, since her general standard is already rock-bottom. During a House Oversight Committee hearing, Greene accused Rep. Jasmine Crockett of...235 comments 271 Recs
Recent Diaries
- KILLER IN MY CLASSROOM? By Don C. Reed The story I am about to tell you is true, subject only to the vagaries of memory… Call him Y. He was a student in my 8th grade English classroom, someone I had known for a long time, because we used to give him...1 comments 0 Recs
- How many enemies of my enemy are my friends is advisable? 1-The well-known one: The well-known canoodling of the Evangelicals and the Israeli Government has been going on for around 50+ years. US evangelicals drive Republican support for Israel [BBC]...4 comments 7 Recs
- We have been investigating which groups are appropriate for republishing Boosting Biden posts on a variety of topics.2 comments 1 Recs
- First the grey elephant in the room: The NYT not only released a ridiculous poll this week, but then they milked it with follow up stories on why Biden is losing so badly and why everyone is ...18 comments 28 Recs
- Good morning, everyone and welcome to Saturday’s Morning Open Thread. Morning Open Thread is a daily, copyrighted post from a host of editors and guest writers. We support our community, invite ...37 comments 12 Recs
- Just a quick one here: I’ve worked in graphic design and Photoshop for over 30 years. I’m certified on LinkedIn as a Phtoshop expert. So I have some basis for my speculation. And something seems off here. It looks like this photo, attributed to Getty,...22 comments 17 Recs
- Welcome to White House Picture of the Day Showcasing a photo from the White House Photography staff, led by Adam Schultz. Feel free to join us anytime and share with us what is happening in ...3 comments 10 Recs
- America’s mayor posted this. It has since been deleted… But the Arizona AG was on the ball. Because less than 30 minutes later….14 comments 53 Recs
- . As you can see by Itzl's concerned look, this group is for us to check in at to let people know we are alive, doing OK, and not affected by such things as heat, blizzards, floods, wild fires,13 comments 8 Recs
- Al Jazeera has a fascinating look at some of the Western volunteers, many veterans of the International Legion fighting against the Russian invasion of Ukraine, who are now making their way to Myanmar to help in the ongoing fight against the military...4 comments 7 Recs
- AMERICAN CANYON, Calif. – On May 14, the California Fish and Game Commission unanimously adopted emergency regulations for Chinook salmon fishing closures in the Central Valley and Klamath River Basins due to continued dramatic population declines. As...1 comments 4 Recs
- Russian forces conducted a limited series of drone strikes against Ukraine on the night of May 16 to 17 and individual missile strikes on May 17. Ukrainian Air Force Commander Lieutenant General Mykola Oleshchuk reported that Russian forces launched 20...11 comments 79 Recs
- There is something I’m really having a hard time wrapping my head around and that is the term “average speed”. I think it’s an absolute misnomer. So, I use the following example. It took me fifty (50) minutes to drive 25 miles. I did the math, and that...6 comments 3 Recs
- Please read and follow instructions before opening. Or consult your physician. �This is a late-night gathering for non-serious palaver, enter at your own risk. Frivolity and silliness are highly ...10 comments 4 Recs
- Welcome to the Overnight News Digest with a crew consisting of founder Magnifico, regular editors side pocket, maggiejean, Chitown Kev, eeff, Magnifico, annetteboardman, Besame, jck, and �...20 comments 25 Recs
- We now have data for three years of COVID death rates. The first year, 2020, was almost over when the vaccine became available on December 13 of 2020. This graph shows how the disease progressed during the two years after vaccines became available. The...31 comments 59 Recs
- This morning’s organizing meeting largely consisted of people phoning in their input, but the trip to N. Charleston City Hall wasn’t wasted. We warned the mayor’s office about the UnityCity scheme and let US Senator Tim Scott know that we knew about it...2 comments 1 Recs
- Kitchen Table Kibitzing is a community series for those who wish to share a virtual kitchen table with other readers of Daily Kos who aren’t throwing pies at one another. Drop by to talk about music, your weather, your garden, or what you cooked for...86 comments 23 Recs
- — — When a prediction methodology becomes obsolete, should society keep relying on it, like it was something more than a roll of the dice? Why polling is so often wrong by James Rosen, bostonglobe.com — April 3, 2024 If political pollsters were...16 comments 32 Recs
- My, my, my, the Manhattan Criminal Courthouse has become the place to see and be seen for the proto-fascist sycophant set. It’s like Studio 54, only for excruciatingly dorky, power-hungry nitwits. “Hey, wanna play hooky from Congress to help an...39 comments 180 Recs