Inscriptions, Al-Jallad explained, tend to cluster on higher ground, where nomadic herders could keep an easier watch for predators. In a landscape with no other traces of human civilization, the rocks preserved the nomads’ names and genealogies, along with descriptions of their animals, their wars, their journeys, and their rituals. There were prayers to deities, worries about the lack of rain, and complaints about the cruelty of Romans.

I wonder what you think of a distinction between the more traditional 'scholar's box', and the proto-databases that were used to write dictionaries and then for projects such as the Mundaneum. I can't help feeling there's a significant difference between a collection of notes meant for a single person, and a collection meant to be used collaboratively. But not sure exactly how to characterize this difference. Seems to me that there's a tradition that ended up with the word processor, and another one that ended up with the database. I feel that the word processor, unlike the database, was a dead end.

research data life cycle

The BTL Online database provides electronic access to all editions of Latin texts published in the Bibliotheca Teubneriana, ranging from antiquity and late antiquity to medieval and neo-Latin texts. A total of approximately 13 million word forms are thus accessible electronically.

ake sure you are mentoring someone, always. Make sure you’re always mentoring others. And I have found that mentoring others gives me new perspectives on things. Because I may be telling them things, but I’m learning a lot while I’m telling them.

Die Erfahrungen Ermans und seiner Mitarbeiter lehren nur zu deutlich, dass die Buchform der Präsentation eines solchen Materialbestands durchaus nicht entgegenkommt.

Ausgangspunkt und Zentrum der Arbeit am Altägyptischen Wörterbuch ist die Anlage eines erschöpfenden Corpus ägyptischer Texte.

9/8b2 "Multiple storage" als Notwendigkeit derSpeicherung von komplexen (komplex auszu-wertenden) Informationen.

Good stories are of diminishing importance — ironic, given how audiences were traditionally drawn into a world, like that of The Odyssey, by way of a single character’s journey.

Keeping track of research materials used to require an excellent memory, exceptional bookkeeping skills or blind luck; now we have databases.

A career-line study of the presidents, all cabinet members,and all members of the Supreme Court. This 1 'already have onIBM cards from the constitutional period through Truman'ssecond term, but I want to expand the items used and analyze itafresh.

How To Choose The Right Database?

Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.

Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain.

In 1896, Dewey formed a partnership with Herman Hollerith and the Tabulating Machine Company (TMC) to provide the punch cards used for the electro-mechanical counting system of the US government census operations. Dewey’s relationship with Hollerith is significant as TMC would be renamed International Business Machines (IBM) in 1924 and become an important force in the information age and creator of the first relational database.

It wasnot until we had completely re-sorted all our innumerable sheets ofpaper according to subjects, thus bringing together all the facts relatingto each, whatever the trade concerned, or the place or the date—andhad shuffled and reshuffled these sheets according to various tentativehypotheses—that a clear, comprehensive and verifiable theory of theworking and results of Trade Unionism emerged in our minds; tobe embodied, after further researches by way of verification, in ourIndustrial Democracy (1897).

On many occasions we have been com¬pelled to break off the writing of a particular chapter, or even of aparticular paragraph, in order to test, by reshuffling the whole of ournotes dealing with a particular subject, a particular place, a particularorganisation or a particular date, the relative validity of hypotheses asto cause and effect. I may remark, parenthetically, that we have foundthis “ game with reality ”, this building up of one hypothesis andknocking it down in favour of others that had been revealed or verifiedby a new shuffle of the notes—especially when we severally “ backed ”rival hypotheses—a most stimulating recreation! In that way alonehave we been able “ to put our bias out of gear ”, and to make ourorder of thought correspond, not with our own prepossessions, butwith the order of things discovered by our investigations.

An instance may be given of the necessity of the “ separate sheet ” system.Among the many sources of information from which we constructed our bookThe Manor and the Borough were the hundreds of reports on particular boroughsmade by the Municipal Corporation Commissioners in 1835 .These four hugevolumes are well arranged and very fully indexed; they were in our own possession;we had read them through more than once; and we had repeatedly consulted themon particular points. We had, in fact, used them as if they had been our own boundnotebooks, thinking that this would suffice. But, in the end, we found ourselvesquite unable to digest and utilise this material until we had written out every oneof the innumerable facts on a separate sheet of paper, so as to allow of the mechanicalabsorption of these sheets among our other notes; of their complete assortment bysubjects; and of their being shuffled and reshuffled to test hypotheses as to suggestedco-existences and sequences.

Perseus, a site, based at Tufts, specializing in Greek and Latin

Create the new empty table Write to both old and new table Copy data (in chunks) from old to new Validate consistency Switch reads to new table Stop writes to the old table Cleanup old table

Explore the Origins and Forced Relocations of Enslaved Africans Across the Atlantic World The SlaveVoyages website is a collaborative digital initiative that compiles and makes publicly accessible records of the largest slave trades in history. Search these records to learn about the broad origins and forced relocations of more than 12 million African people who were sent across the Atlantic in slave ships, and hundreds of thousands more who were trafficked within the Americas. Explore where they were taken, the numerous rebellions that occurred, the horrific loss of life during the voyages, the identities and nationalities of the perpetrators, and much more.

It is thanks to decades of painstaking, difficult work that we know a great deal about the scale of human trafficking across the Atlantic Ocean and about the people aboard each ship. Much of that research is available to the public in the form of the SlaveVoyages database. A detailed repository of information on individual ships, individual voyages and even individual people, it is a groundbreaking tool for scholars of slavery, the slave trade and the Atlantic world. And it continues to grow. Last year, the team behind SlaveVoyages introduced a new data set with information on the domestic slave trade within the United States, titled “Oceans of Kinfolk.”

databases is an async SQL query builder that works on top of the SQLAlchemy Core expression language.

This is where off-site backups come into play. For this purpose, I recommend Borg backup. It has sophisticated features for compression and encryption, and allows you to mount any version of your backups as a filesystem to recover the data from. Set this up on a cronjob as well for as frequently as you feel the need to make backups, and send them off-site to another location, which itself should have storage facilities following the rest of the recommendations from this article. Set up another cronjob to run borg check and send you the results on a schedule, so that their conspicuous absence may indicate that something fishy is going on. I also use Prometheus with Pushgateway to make a note every time that a backup is run, and set up an alarm which goes off if the backup age exceeds 48 hours. I also have periodic test alarms, so that the alert manager’s own failures are noticed.

RAID is complicated, and getting it right is difficult. You don’t want to wait until your drives are failing to learn about a gap in your understanding of RAID. For this reason, I recommend ZFS to most. It automatically makes good decisions for you with respect to mirroring and parity, and gracefully handles rebuilds, sudden power loss, and other failures. It also has features which are helpful for other failure modes, like snapshots. Set up Zed to email you reports from ZFS. Zed has a debug mode, which will send you emails even for working disks — I recommend leaving this on, so that their conspicuous absence might alert you to a problem with the monitoring mechanism. Set up a cronjob to do monthly scrubs and review the Zed reports when they arrive. ZFS snapshots are cheap - set up a cronjob to take one every 5 minutes, perhaps with zfs-auto-snapshot.

these days hardware RAID is almost always a mistake. Most operating systems have software RAID implementations which can achieve the same results without a dedicated RAID card.

Failing disks can show signs of it in advance — degraded performance, or via S.M.A.R.T reports. Learn the tools for monitoring your storage medium, such as smartmontools, and set it up to report failures to you (and test the mechanisms by which the failures are reported to you).

RAID gets more creative with three or more hard drives, utilizing parity, which allows it to reconstruct the contents of failed hard drives from still-online drives.

A more reliable solution is to store the data on a hard drive1. However, hard drives are rated for a limited number of read/write cycles, and can be expected to fail eventually.

The worst way I can think of is to store it on a microSD card. These fail a lot. I couldn’t find any hard data, but anecdotally, 4 out of 5 microSD cards I’ve used have experienced failures resulting in permanent data loss.

The console is a killer SQLite feature for data analysis: more powerful than Excel and more simple than pandas. One can import CSV data with a single command, the table is created automatically

This is not a problem if your DBMS supports SQL recursion: lots of data can be generated with a single query. The WITH RECURSIVE clause comes to the rescue.

Queries became impractically slow around the 500,000 cell mark, but were still below 2 seconds for a 100,000 cell query. Therefore, if you anticipate a dataset larger than a few hundred thousand cells, it would probably be smart to choose a more scalable option.

DuckDB is an embeddable SQL OLAP database management system

The Splitgraph DDN is a single SQL endpoint that lets you query over 40,000 public datasets hosted on or proxied by Splitgraph.You can connect to it from most PostgreSQL clients and BI tools without having to install anything else. It supports all read-only SQL constructs, including filters and aggregations. It even lets you run joins across distinct datasets.

Archive your AWS data to reduce storage cost

So in brief, for our application service, if we understand the access patterns very well, they’re repeatable, they’re consistent, and scalability is a big factor, then NoSQL is a perfect choice.

Comparison Time … 🤞

Which database technology to choose

From a narratological perspective, it would probably be fair to say that most databases are tragic. In their design, the configuration of their user interfaces, the selection of their contents, and the indexes that manage their workings, most databases are limited when set against the full scope of the field of information they seek to map and the knowledge of the people who created them. In creating a database, we fight against the constraints of the universe – the categories we use to sort out the world; the limitations of time and money and technology – and succumb to them.

Finding a database management system that works for you

I’m sharing a few insights I specifically found useful for developers who are not specialized in this domain.

1) Redash and Falcon focus on people that want to do visualizations on top of SQL2) Superset, Tableau and PowerBI focus on people that want to do visualizations with a UI3) Metabase and SeekTable focus on people that want to do quick analysis (they are the closest to an Excel replacement)

supporting this field is extremely easy If you keep raw data, it's just a matter of adding a getter method to the Article class.

@property
def highlights(self) -> Sequence[Highlight]:
    default = [] # defensive to handle older export formats that had no annotations
    jsons = self.json.get('annotations', default)
    return list(map(Highlight, jsons))

query language doesn't necessarily mean a database. E.g. see pandas which is capable of what SQL is capable of, and even more convenient than SQL for our data exploration purposes.

cachew lets you cache function calls into an sqlite database on your disk in a matter of single decorator (similar to functools.lru_cache). The difference from functools.lru_cache is that cached data is persisted between program runs, so next time you call your function, it will only be a matter of reading from the cache.

Imagine that you're using a database to export them, so your schema is: TABLE Article(STRING id, STRING url, STRING title, DATETIME added). One day, the developers expose highlights (or annotations) from the private API and your export script stats receiving it in the response JSON. It's quite useful data to have! However, your database can't just magically change to conform to the new field.

Storage saved by using a database instead of plaintext is marginal and not worth the effort.

if necessary use databases as an intermediate layer to speed access up and as an additional interface to your data Nothing wrong with using databases for caching if you need it!

I want to argue very strongly against forcing the data in the database, unless it's really inevitable.

FKs don't work well with online schema migrations.

FKs are a performance impact. The fact they require indexes is likely fine, since those indexes are needed anyhow. But the lookup made for each insert/delete is an overhead.

FKs are in your way to shard your database. Your app is accustomed to rely on FK to maintain integrity, instead of doing it on its own. It may even rely on FK to cascade deletes (shudder). When eventually you want to shard or extract data out, you need to change & test the app to an unknown extent.

To address the availability concern, new architectures were developed to minimize the impact of partitions. For instance, splitting data sets into smaller ranges called shards can minimize the amount of data that is unavailable during partitions. Furthermore, mechanisms to automatically alter the roles of various cluster members based on network conditions allow them to regain availability quickly

typically less flexible and generalized than their more conventional relational counterparts. They also usually only offer a subset of full SQL and relational features, which means that they might not be able to handle certain kinds of usage. Many NewSQL implementations also store a large part of or their entire dataset in the computer's main memory. This improves performance at the cost of greater risk to unpersisted changes

using a mixture of different database types is the best approach for handling the data of your projects

best suited for use cases with high volumes of relational data in distributed, cloud-like environments

CAP theorem is a statement about the trade offs that distributed databases must make between availability and consistency. It asserts that in the event of a network partition, a distributed database can choose either to remain available or remain consistent, but it cannot do both. Cluster members in a partitioned network can continue operating, leading to at least temporary inconsistency. Alternatively, at least some of the disconnected members must refuse to alter their data during the partition to ensure data consistency

NewSQL databases: bringing modern scalability and performance to the traditional relational pattern

aggregate queries like summing, averaging, and other analytics-oriented processes can be difficult or impossible

Column-family databases are good when working with applications that requires great performance for row-based operations and highly scalability

it helps to think of column family databases as key-value databases where each key (row identifier) returns a dictionary of arbitrary attributes and their values (the column names and their values)

Column-family databases: databases with flexible columns to bridge the gap between relational and document databases

querying for the connection between two users of a social media site in a relational database is likely to require multiple table joins and therefore be rather resource intensive. This same query would be straightforward in a graph database that directly maps connections

Graph databases are most useful when working with data where the relationships or connections are highly important

network databases require step-by-step traversal to travel between items and are limited in the types of relationships they can represent.

Graph databases: mapping relationships by focusing on how connections between data are meaningful

Document databases: Storing all of an item's data in flexible, self-describing structures

Document databases are a good choice for rapid development because you can change the properties of the data you want to save at any point without altering existing structures or data. You only need to backfill records if you want to. Each document within the database stands on its own with its own system of organization. If you're still figuring out your data structure and your data is mainly composed discrete entries that don't include a lot of cross references, a document database might be a good place to start. Be careful, however, as the extra flexibility means that you are responsible for maintaining the consistency and structure of your data, which can be extremely challenging

Though the data within documents is organized within a structure, document databases do not prescribe any specific format or schema

Key-value stores are often used to store configuration data, state information, and any data that might be represented by a dictionary or hash in a programming language. Key-value stores provide fast, low-complexity access to this type of data

Key-value databases: simple, dictionary-style lookups for basic storage and retrieval

NoSQL databases: modern alternatives for data that doesn't fit the relational paradigm

relational databases are often a good fit for any data that is regular, predictable, and benefits from the ability to flexibly compose information in various formats. Because relational databases work off of a schema, it can be more challenging to alter the structure of data after it is in the system. However, the schema also helps enforce the integrity of the data, making sure values match the expected formats, and that required information is included. Overall, relational databases are a solid choice for many applications because applications often generate well-ordered, structured data

querying language called SQL, or structured query language, was created to access and manipulate data stored with that format

Relational databases: working with tables as a standard solution to organize well-structured data

database schema is a description of the logical structure of a database or the elements it contains. Schemas often include declarations for the structure of individual entries, groups of entries, and the individual attributes that database entries are comprised of. These may also define data types and additional constraints to control the type of data that may be added to the structure

Network databases: mapping more flexible connections with non-hierarchical links

Hierarchical databases: using parent-child relationships to map data into trees

Hierarchical databases are not used much today due to their limited ability to organize most data and because of the overhead of accessing data by traversing the hierarchy

The first flat file databases represented information in regular, machine parse-able structures within files. Data is stored in plain text, which limits the type of content that can be represented within the database itself. Sometimes, a special character or other indicator is chosen to use as a delimiter, or marker for when one field ends and the next begins. For example, a comma is used in CSV (comma-separated values) files, while colons or white-space are used in many data files in Unix-like systems

Some advantages of this format

While flat file databases are simple, they are very limited in the level of complexity they can handle

Elec­tronic Recording Machine Accounting (ERMA)

MySQL’s replication architecture means that if bugs do cause table corruption, the problem is unlikely to cause a catastrophic failure.

The bug we ran into only affected certain releases of Postgres 9.2 and has been fixed for a long time now. However, we still find it worrisome that this class of bug can happen at all. A new version of Postgres could be released at any time that has a bug of this nature, and because of the way replication works, this issue has the potential to spread into all of the databases in a replication hierarchy.

If the RRID is well-formed, and if the lookup found the right record, a human validator tags it a valid RRID — one that can now be associated mechanically with occurrences of the same resource in other contexts. If the RRID is not well-formed, or if the lookup fails to find the right record, a human validator tags the annotation as an exception and can discuss with others how to handle it. If an RRID is just missing, the validator notes that with another kind of exception tag.

Giga Science Database

Data gathering is ubiquitous in science. Giant databases are currently being minedfor unknown patterns, but in fact there are many (many) known patterns that simplyhave not been catalogued. Consider the well-known case of medical records. A patient’smedical history is often known by various individual doctor-offices but quite inadequatelyshared between them. Sharing medical records often means faxing a hand-written noteor a filled-in house-created form between offices.

I will use a mathematical tool calledologs, or ontology logs, to givesome structure to the kinds of ideas that are often communicated in pictures like theone on the cover. Each olog inherently offers a framework in which to record data aboutthe subject. More precisely it encompasses adatabase schema, which means a system ofinterconnected tables that are initially empty but into which data can be entered.

Supported sites

This in turn means that Redis Cluster does not have to take meta data in the data structures in order to attempt a value merge, and that the fancy commands and data structures supported by Redis are also supported by Redis Cluster. So no additional memory overhead, no API limits, no limits in the amount of elements a value can contain, but less safety during partitions.

Fast restart. If a server is temporarily taken down, this capability restores the index from a saved copy, eliminating delays due to index rebuilding.

Unlike other databases that use the linux file system that was built for rotational drives, Aerospike has implemented a log structured file system to access flash – raw blocks on SSDs – directly.