Interview Q and A for Cassandra DB Part - 4

 

Why can’t I set listen_address to listen on 0.0.0.0 (all my addresses)?

Cassandra is a gossip-based distributed system and listen_address is the address a node tells other nodes to reach it at. Telling other nodes “contact me on any of my addresses” is a bad idea; if different nodes in the cluster pick different addresses for you, Bad Things happen.

If you don’t want to manually specify an IP to listen_address for each node in your cluster (understandable!), leave it blank and Cassandra will use InetAddress.getLocalHost() to pick an address. Then it’s up to you or your ops team to make things resolve correctly (/etc/hosts/, dns, etc).

One exception to this process is JMX, which by default binds to 0.0.0.0 (Java bug 6425769).

 

I delete data from Cassandra, but disk usage stays the same. What gives?

Data you write to Cassandra gets persisted to SSTables. Since SSTables are immutable, the data can’t actually be removed when you perform a delete, instead, a marker (also called a “tombstone”) is written to indicate the value’s new status. Never fear though, on the first compaction that occurs between the data and the tombstone, the data will be expunged completely and the corresponding disk space recovered. See Compaction for more detail.

 

 

Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?

This happens when you have the same token assigned to each node. Don’t do that.

Most often this bites people who deploy by installing Cassandra on a VM (especially when using the Debian package, which auto-starts Cassandra after installation, thus generating and saving a token), then cloning that VM to other nodes.

The easiest fix is to wipe the data and commitlog directories, thus making sure that each node will generate a random token on the next restart.

 

 

Can I change the replication factor (a a keyspace) on a live cluster?

Yes, but it will require running a full repair (or cleanup) to change the replica count of existing data:

• Alter the replication factor for desired keyspace (using cqlsh for instance).
• If you’re reducing the replication factor, run nodetool cleanup on the cluster to remove surplus replicated data. Cleanup runs on a per-node basis.
• If you’re increasing the replication factor, run nodetool repair -full to ensure data is replicated according to the new configuration. Repair runs on a per-replica set basis. This is an intensive process that may result in adverse cluster performance. It’s highly recommended to do rolling repairs, as an attempt to repair the entire cluster at once will most likely swamp it. Note that you will need to run a full repair (-full) to make sure that already repaired sstables are not skipped.

Can I Store (large) BLOBs in Cassandra?

Cassandra isn’t optimized for large file or BLOB storage and a single blob value is always read and send to the client entirely. As such, storing small blobs (less than single digit MB) should not be a problem, but it is advised to manually split large blobs into smaller chunks.

Please note in particular that by default, any value greater than 16MB will be rejected by Cassandra due the max_mutation_size_in_kb configuration of the Cassandra Configuration File file (which default to half of commitlog_segment_size_in_mb, which itself default to 32MB).

 

 

Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?

Nodetool relies on JMX, which in turn relies on RMI, which in turn sets up its own listeners and connectors as needed on each end of the exchange. Normally all of this happens behind the scenes transparently, but incorrect name resolution for either the host connecting, or the one being connected to, can result in crossed wires and confusing exceptions.

If you are not using DNS, then make sure that your /etc/hosts files are accurate on both ends. If that fails, try setting the -Djava.rmi.server.hostname=<public name> JVM option near the bottom of cassandra-env.sh to an interface that you can reach from the remote machine.

 

 

Will batching my operations speed up my bulk load?

No. Using batches to load data will generally just add “spikes” of latency. Use asynchronous INSERTs instead, or use true Bulk Loading.

An exception is batching updates to a single partition, which can be a Good Thing (as long as the size of a single batch stay reasonable). But never ever blindly batch everything!

On RHEL nodes are unable to join the ring

Check if SELinux is on; if it is, turn it off.

 

 

Why does top report that Cassandra is using a lot more memory than the Java heap max?

Cassandra uses Memory Mapped Files (mmap) internally. That is, we use the operating system’s virtual memory system to map a number of on-disk files into the Cassandra process’ address space. This will “use” virtual memory; i.e. address space, and will be reported by tools like top accordingly, but on 64 bit systems virtual address space is effectively unlimited so you should not worry about that.

What matters from the perspective of “memory use” in the sense as it is normally meant, is the amount of data allocated on brk() or mmap’d /dev/zero, which represent real memory used. The key issue is that for a mmap’d file, there is never a need to retain the data resident in physical memory. Thus, whatever you do keep resident in physical memory is essentially just there as a cache, in the same way as normal I/O will cause the kernel page cache to retain data that you read/write.

The difference between normal I/O and mmap() is that in the mmap() case the memory is actually mapped to the process, thus affecting the virtual size as reported by top. The main argument for using mmap() instead of standard I/O is the fact that reading entails just touching memory - in the case of the memory being resident, you just read it - you don’t even take a page fault (so no overhead in entering the kernel and doing a semi-context switch). This is covered in more detail here.

What are seeds?

Seeds are used during startup to discover the cluster.

If you configure your nodes to refer some node as seed, nodes in your ring tend to send Gossip message to seeds more often (also see the section on gossip) than to non-seeds. In other words, seeds are worked as hubs of Gossip network. With seeds, each node can detect status changes of other nodes quickly.

Seeds are also referred by new nodes on bootstrap to learn other nodes in ring. When you add a new node to ring, you need to specify at least one live seed to contact. Once a node join the ring, it learns about the other nodes, so it doesn’t need seed on subsequent boot.

You can make a seed a node at any time. There is nothing special about seed nodes. If you list the node in seed list it is a seed

Seeds do not auto bootstrap (i.e. if a node has itself in its seed list it will not automatically transfer data to itself) If you want a node to do that, bootstrap it first and then add it to seeds later. If you have no data (new install) you do not have to worry about bootstrap at all.

Recommended usage of seeds:

• pick two (or more) nodes per data center as seed nodes.
• sync the seed list to all your nodes

 

 

 

Does single seed mean single point of failure?

The ring can operate or boot without a seed; however, you will not be able to add new nodes to the cluster. It is recommended to configure multiple seeds in production system.

Why can’t I call jmx method X on jconsole?

Some of JMX operations use array argument and as jconsole doesn’t support array argument, those operations can’t be called with jconsole (the buttons are inactive for them). You need to write a JMX client to call such operations or need array-capable JMX monitoring tool.

 

 

 

Why do I see “… messages dropped …” in the logs?

This is a symptom of load shedding – Cassandra defending itself against more requests than it can handle.

Internode messages which are received by a node, but do not get not to be processed within their proper timeout (see read_request_timeout, write_request_timeout, … in the Cassandra Configuration File), are dropped rather than processed (since the as the coordinator node will no longer be waiting for a response).

For writes, this means that the mutation was not applied to all replicas it was sent to. The inconsistency will be repaired by read repair, hints or a manual repair. The write operation may also have timeouted as a result.

For reads, this means a read request may not have completed.

Load shedding is part of the Cassandra architecture, if this is a persistent issue it is generally a sign of an overloaded node or cluster.

Cassandra dies with java.lang.OutOfMemoryError: Map failed

If Cassandra is dying specifically with the “Map failed” message, it means the OS is denying java the ability to lock more memory. In linux, this typically means memlock is limited. Check /proc/<pid of cassandra>/limits to verify this and raise it (eg, via ulimit in bash). You may also need to increase vm.max_map_count. Note that the debian package handles this for you automatically.

What happens if two updates are made with the same timestamp?

Updates must be commutative, since they may arrive in different orders on different replicas. As long as Cassandra has a deterministic way to pick the winner (in a timestamp tie), the one selected is as valid as any other, and the specifics should be treated as an implementation detail. That said, in the case of a timestamp tie, Cassandra follows two rules: first, deletes take precedence over inserts/updates. Second, if there are two updates, the one with the lexically larger value is selected.

Why bootstrapping a new node fails with a “Stream failed” error?

Two main possibilities:

1. the GC may be creating long pauses disrupting the streaming process
2. compactions happening in the background hold streaming long enough that the TCP connection fails

In the first case, regular GC tuning advices apply. In the second case, you need to set TCP keepalive to a lower value (default is very high on Linux). Try to just run the following:

$ sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5

To make those settings permanent, add them to your /etc/sysctl.conf file.

Note: GCE’s firewall will always interrupt TCP connections that are inactive for more than 10 min. Running the above command is highly recommended in that environment.

Q18) Explain what is logging in Cassandra.

Answer:
Each and every log list is stored in logback.xml and you can change any property indiscriminately adopting this file. INFO level logs are written to system.log debug.log include debugging level entries.

 

 

Q19) Is HDFS utilized in Cassandra? If yes, where?

Answer:
The adaptable medium of Cassandra file system is termed as HDFS. It enables to operate on huge data with comfort. It has successfully replaced oriental methods of dealing with large quantum of nodal structures.

Q20) Explain the fundamental difference between Cassandra and Hadoop.

Answer:
Hadoop
It is an analytical dashboard that helps manage a large database.
Cassandra
Cassandra is purely NoSQL Database which enables high-speed performance together with executing online trAnswer:action volumes.

Q21) Explain the rudimentary difference between Cassandra and HBase.

Answer:
HBase
It is a multiple discharged database developed on Java and by nature an open source and NoSQL centric. It is used to exemplify online applications.
Cassandra
It’s an operative structure almost resembles HBase but the functionality is better served and attached to it are many advanced features as it was created post observing the fallacies of HBase. So, many technical loopholes have been eliminated, that were a part of HBase.

 

 

Q22) Elaborate on Identifiers.

Answer:
These are tokens, using usual expressions. These tokens directly relate to recognizing specific columns, tables and objects.

 

7) What are the main components of Cassandra?

The components of Cassandra include:

·       Node

·       Data cluster

·       Commit log

·       Cluster

·       Meme-table

·       SSTable 

·       Bloom filter

 

 

8) What are the functions of Cassandra?

Ans: This database supports two main categories of functions:

Scalar functions: Its primary purpose is taking some groups of values and producing an output with it. 

Aggregate functions: Its primary function is producing a combined result using selected multiple rows.

 

15) How is data distribution done?

Ans: Cassandra database is a highly-available database, and it stores data by evenly dividing the data around its nodes. For this, it uses the Murmur3 partitioning function to distribute given data in nodes evenly. 

 

 

16) How does Cassandra store data?

Ans: The data storage path in Cassandra begins with the memtable where the data is stored temporarily and is also called a commit log. And once committed, the data is periodically flushed and written into SSTable 

 

 

 

18) What is a direct request?

Ans: Direct request in Cassandra is a part of the read operation. In this, the coordinator node contacts the replica node.

19) Define digest request?

Ans: When the coordinator node contacts replicas, it actually requests those nodes which reply fastest. Then these contacted nodes respond with a digest of the data required.

 

 

 

20) Explain read repair request?

Ans: When the coordinator node sends requests, it checks in the nodes for any outdated data. This data is sent for a background read and repair and is replaced with the updated data. Read and repair requests, is a method to keep the data updated, and it also makes sure that the requested row is consistent on all replicas. 

 

 

21) What is a write operation? 

Ans: There is step by step operations in writing, which goes as follows. 

Step1: It is as soon as it receives its request it sends the data to the commit log to save the data.

Step2: Data is inserted upon request and then sent to commit log to save data. 

Step3: If the memtable reaches its limit then data is flushed to SSTable.

 

23) What do you mean by ACID?

Ans: ACID stands for

Atomicity: Which means either your transaction can fail or commit

Consistency: Its definition changes from software to software or an application to application, but its general meaning is that data has to stay consistent.

Isolation: Data has to be isolated and separated from each other 

Durability: It assures you that once the database receives data, it should ensure that the data is processed. So it is an advantage if the database fails, then the data will not be lost.

 

 

24) What is BASE?

Ans: Not every application or software needs this strong consistency, so this is where base comes into action. The BASE stands for Basically Available Soft-state Eventually-consistent properties.NoSQL databases basically use these models.

 

 

27) What are the best monitor tools of Cassandra?

Ans: Although Cassandra comes with built-in tolerance features, it still needs to be monitored for effective results. Here are some tools which Cassandra uses to monitor its databases:

·       Solarwind server and application monitor

·       Instana

·       Instaclustr

·       AppDynamics

·       Dynatrace

·       Machine engine applications manager.

 

32) Name the key roles of CQL?

It is very necessary to provide different types of users with different kinds of roles depending upon their requirements. It ensures the security of database users. and their key roles goes as follows:

·       Create a role

·       Alter role

·       Drop role

·       Grant role

·       Revoke role

·       List role

 

70.Do I need to use a caching layer (like memcached) with Cassandra?

Cassandra negates the need for extra software caching layers like memcached through its distributed architecture, fast write throughput capabilities, and internal memory caching structures.

 

 

9). List out some key features of Apache Cassandra?
Ans: It is scalable, fault-tolerant, and consistent.
It is a column-oriented database.
Its distribution design is based on Amazon’s Dynamo and its data model on Google’s Bigtable.
Created at Facebook, it differs sharply from relational database management systems.
Cassandra implements a Dynamo-style replication model with no single point of failure, but adds a more powerful “column family” data model.
Cassandra is being used by some of the biggest companies such as Facebook, Twitter, Cisco, Rackspace, ebay, Twitter, Netflix, and more.

 

 

 

Links: - 

 

http://www.interviewquestionspdf.com/2015/11/top-50-cassandra-interview-questions.html

https://cassandra.apache.org/doc/latest/faq/

https://mindmajix.com/cassandra-interview-questions-and-answers

https://www.gangboard.com/blog/cassandra-interview-questions-and-answers

https://www.wisdomjobs.com/e-university/apache-cassandra-interview-questions.html

 

 

 

http://geoinsyssoft.com/cassandra-interview-questions/

 

 

http://crackyourinterview.com/Ads-Cassandra-Interview-Questions-Part1.aspx

 

 

http://pkkjava.blogspot.com/p/cassandra-interview-questions.html

 

Interview Q and A for Cassandra DB Part - 3

 181. Starting Cassandra 

Bin/Cassandra

Cassandra writes several log messages 

“ State jump to normal”   will   be final message at start 

 

182. Installing and starting Cassandra from a tarball requires extensive setup and configuration.

False

 

183. You must first obtain a license before using DataStax Enteprise edition for development.

False

 

184. What is a repair   ?

Repair is a deliberate action to cope with cluster entropy

Entropy can arise from nodes that were down longer than the hint window, dropped mutations, or other causes 

A repair operates on all of the nodes in replica set by default 

Ensures that all replicas have identical copies of a given partition

Consists of two phases : 

  Build Markle tree of the data per partition

 Replicas then compare the differences between their trees and stream the differences to each other as needed.

 

185. Markle tree exchange ?

Start with the root of the tree ( a list of one hash value)

The origin sends the list of hashes at the current level 

The destination diffs the list of hashes against its own, then requests subtree that are different 

If there are no differences, the request can terminate

Repeat steps 2 and 3 until leaf nodes are reached 

The origin sends the values of the keys in the resulting set 

 

186. Why is repair necessary ?

A node’s data can get inconsistent over time ( Repair is just a maintenance action in this case)

If a node goes down for some time, it misses writes and will need to catch up 

Sometimes it is best to repair a node : 

 

 If the node has been down longer than the length specified in MAX_HINT_WINDOW_IN_MS, the node is out of sync.

Depending on amount of data, might be faster to repair 

 

Sometimes it is better to bring the node back as a new node  :

If there is a significant amount of data, might be faster just to bring in a new node and stream data just to that node 

 

187. What are incremental repairs? 

To avoid the need for constant tree construction, incremental repairs have been introduced 

Idea is to persist already repaired data, and only calculate merkle trees for sstables that haven’t previously undergone repairs 

This allow the repair process to stay performant and lightweight

 

188. What are big data systems?

 Applications involving the "three V's"

•  Volume: gigabytes, growing to terabytes and beyond

•  Velocity: sensor data, click streams, financial transactions

•  Variety: data must be ingested from many different formats

 Characteristics requiring 

•  multi-region availability

•  very fast and reliable response 

•  no single point of failure

 

189. What strategies help manage big data?

•  Distribute data across nodes

•  Relax consistency requirements

•  Relax schema requirements

•  Optimize data to suit actual needs

 

190. What is the NoSQL landscape?

  Four broad classes of non-relational database

•  Graph: data elements each relate to n others in a graph/network

•  Key-Value: keys map to arbitrary values of any data type

•  Document: document sets (JSON) queryable in whole or part

•  Column Family: keys mapped to sets of n-number of typed columns 

 Three key factors help navigate the landscape

•  Consistency: do you get identical results, regardless which node is queried?

 •  Availability: can the cluster respond to very high write and read volumes?

•  Partition Tolerance: is the cluster still available when part of it goes dark?

 

191. What is Cassandra?

Massively linearly scalable NoSQL database

•  Fully distributed, with no single point of failure

•  Free and open source, with deep developer support

•  Highly performant, with near-linear horizontal scaling in proper use cases

• Fully peer-to-peer—no master/slave architecture 
• Data center aware 

No single point of failure, due to horizontal scaling

•  horizontal scaling: add commodity hardware to a cluster

•  vertical scaling: add RAM and CPUs to a specialized high performance box

 

192. When is Cassandra the best solution?

Cassandra excels when you need

•  No single point of failure

•  Real-time writes with live operational data analysis

•  Flexible, easily altered data models

•  Near-linear horizontal scaling across commodity servers 

•  Reliable replication across distributed data centers

•  Clearly defined table schema in a NoSQL environment

 

193. When is Cassandra not the best solution?

Traditional RDBMS excels when you need

•  ACID-compliant transactions, with rollback (e.g., bank transfers)

 •  Justification for high-end hardware

 

194. What are Keyspaces? 

•  A namespace for tables in a cluster 

•  All data will reside in some keyspace 

•  Main function: Control replication 

•  Data with different replication requirements will be in different keyspaces

  •  Somewhat analogous to a schema in the relational model 

 

Replication Factor (RF) and replication strategy specified when creating a keyspace 

•  They may be changed later 

•  Below is the CQL (Cassandra Query Language) to create a keyspace 

•  It uses SimpleStrategy and has an RF of 1 

•  You can change the RF using the ALTER KEYSPACE command

 

CREATE KEYSPACE stockwatcher WITH REPLICATION =

{'class' : 'SimpleStrategy', 'replication_factor': 1};

195. What are tables? 

Tables store data in a Cassandra database 

•  Define columns and their metadata (e.g. data type) 

•  Analogous to a relational table 

•  Were called Column Families in the Thrift API 

• CREATE TABLE is used in CQL to define a table 

•  A table must reside in a keyspace 

•  Either selected with USE, or as part of the name of the table, e.g. stockwatcher.user 

•  A table must specify a primary key 

USE stockwatcher;  // Execute any time before the CREATE

CREATE TABLE user ( user)

 

196. What is write operation ?

All writes for a row (including inserts/updates/deletes) are done atomically and in isolation 

• Inserting or updating (multiple) columns in a row is one write operation 

INSERT is Always UPSERT 

• An insert with an existing primary key becomes an update 

• Cassandra will just write the new column value(s) provided 
• Each column inserted will supersede any older values 

• For two concurrent writes with the same primary key, the last write wins 

• i.e. the last write to finish will be returned in subsequent queries 

 

197. What is a cluster? 

• A peer to peer set of nodes 
• Node – one Cassandra instance 
• Rack – a logical set of nodes 
• Data Center – a logical set of racks 

• Cluster – a ring of nodes 

 

198. What are Configuration files ?

• cassandra.yaml 
• One file per node, must agree with other node’s files • Parameters defined throughout 

• cassandra-env.sh • Memory settings • JMX settings 

• log4j-server.properties • Error log settings 

 

199. What key properties are set in cassandra.yaml? 

• cluster_name (default: 'Test Cluster') 

• All nodes in a cluster must have the same value. 

• listen_address (default: localhost) 

• Defines the network interface for gossip connections 

• rpc_address

• Network interface for client connections (0.0.0.0 means all interfaces) 

• rpc_port (default: 9160) 

• port for Thrift client connections

• native_transport_port (default: 9042) 

• port on which CQL native transport listens for clients 

commitlog_directory (default: /var/lib/cassandra/commitlog) 

• Best practice to mount on a separate disk in production (unless SSD) 

• data_file_directories (default: /var/lib/cassandra/data) 

• List of storage directories for data tables (SSTables) 
• saved_caches_directory (default: /var/lib/cassandra/saved_caches) 

• Storage directory for key and row caches 

 

200. What key properties are set in cassandra-env.sh? 

• JVM Heap Size settings 

• MAX_HEAP_SIZE="value" 

• Maximum recommended in production is currently 8G due to current limitations in Java garbage collection 

• HEAP_NEWSIZE="value" 
• Generally set to 1⁄4 of MAX_HEAP_SIZE 

• This file computes the default values, but you can override them as necessary.

 

201. What key properties are set in log4j-server.properties? 

• Cassandra system.log location 
• Default location is /var/log/cassandra/system.log 

• system.log is numerically renamed as it grows over time 

• Cassandra logging level 
• Default logging level is INFO

 

202. What is a node? 

•  Single node database 

•  rpc_address—used to setup how clients come into a cluster 

•  rpc_port (9160)—thrift, how node talks to the application 

•  native_transport_port (9042)—native connections 

•  default—localhost which means you are stuck with only local clients 

 • 0.0.0.0 means clients can come in from anywhere 

 

203. What is a cluster? 

• Nodes join a cluster based on the configuration of their own conf/cassandra.yaml file 

• Key settings include 
• cluster_name – shared name to 

logically distinguish a set of nodes 

• seeds – IP addresses of initial nodes for a new node to contact and discover the cluster topology (best practice to use the same two per data center) 

• listen_address – IP address to determine adaptor through which this particular node communicates to other nodes

204. Where does my data go? 

• Cassandra automatically shards your data. 
• It will put one or more copies of your data on your nodes. 

 

205. What is consistent hashing? 

• Data is stored on nodes in partitions, each identified by a partition key. 

•  Partition – a storage location on a node (analogous to a "table row") 

•  Token – 64 bit integer, generated by a hashing algorithm, identifying a partition's location within a cluster 

• The 264 value token range for a cluster is used as a single ring 

•  So, any partition in a cluster is locatable from one consistent set of hash values, regardless of its node 

•  Specific token range varies by choice of partitioner 

•  Partitioner options discussed ahead 

 

206. What is the partitioner? 

• A system on each node which hashes keys to create a token from designated values in rows being added 

• Hash function – converts a variable length value to a corresponding 
fixed length value 

·       Various partitioners available 

Imagine a 0 to 100 token range (instead of -263 to +263) 

• Each node is assigned a token, just like each of its partitions 

• Node tokens are the highest value in the segment owned by that node 

• This segment is the primary token range of replicas owned by this node 

• Nodes also store replicas keyed to tokens outside this range ("secondary range") 

 

207. How does a partitioner work? 

• A node's partitioner hashes a token 100 from the partition key value of a write request 

• First replica written to node that owns the primary range for this token 

The primary key of a table  determines its partition key values 

Partitioner 

Token 91 'Orange:Oscar' 

 

208. How is data replicated among nodes? 

• SimpleStrategy – create replicas on nodes subsequent  to the primary range node 

CREATE KEYSPACE demo WITH REPLICATION = {'class':'SimpleStrategy', 'replication_factor':3} 
• replication factor of 3 is a recommended minimum 

 

209. is data replicated between data centers? 

• NetworkTopologyStrategy – distribute replicas across racks and data centers 

CREATE KEYSPACE demo WITH REPLICATION = {'class':'NetworkTopologyStrategy', 

'dc-east':2, 'dc-west’:3} 

 

210. What are virtual nodes? 

• Multiple smaller primary range segments – virtual nodes – can be owned by each machine, instead of one larger range 

•  virtual nodes behave like a regular node 

•  available in Cassandra 1.2+

•  default is 256 per machine 

•  not available on nodes combining Cassandra with Solr or Hadoop 

 

211. How are virtual nodes helpful? 

•  token ranges are distributed, so machines bootstrap faster 

•  impact of virtual node failure is spread across entire cluster 

•  token range assignment automated 

Virtual nodes are enabled in cassandra.yaml

• partitions, regular nodes, and virtual nodes are each identified by a token 

• regular or virtual node tokens are the highest value in one segment of the total token range for a cluster, which is the primary range of that node

 

212. What is a coordinator? 

• The node chosen by the client to receive a particular read or write request to its cluster 

• Any node can coordinate any request 

Each client request may be coordinated by a different node 

 

Coordinator Nodes: Its a node which receive the request from client and send the request to the actual node[hash(key) => token] depending upon the token. So all the nodes acts as coordinator node,because every node can receive a request and proxy that request.

 

 

213. What is No single point of failure 

This principle is fundamental to Cassandra's architecture 

 

214. How are client requests coordinated? 

•  The Cassandra driver chooses the node to which each read or write request is sent 

•  Client library providing APIs to manage client read/write requests 

•  Round-robin pattern by default 

The coordinator manages the Replication Factor (RF) 

• Replication factor (RF) – onto how many nodes should a write be copied? 

•  Possible values range from 1 to the total of planned nodes for the cluster 

•  RF is set for an entire keyspace, or for each data center, if multiple 

·       Every write to every node is individually time-stamped 

·       The coordinator also applies the Consistency Level (CL) 

• Consistency level (CL) – how many nodes must acknowledge a read or write request 

•  CL may vary for each request 

•  On success, coordinator notifies client 

• Possible consistency levels include 

• ONE 
• QUORUM(RF/2)+1 

• ALL 

 

215. What is consistency? 

Consistency means to synchronize and how up-to-date a row of Cassandra data is on all of its replicas.

 

• The partition key determines which nodes are sent any given request 

• Consistency Level – sets how many of the nodes to be sent a given request must acknowledge that request, for a response to be returned to the client 

• The meaning varies by type 
• Write request – how many nodes must acknowledge they received and wrote the write request? • Read request – how many nodes must acknowledge by sending their most recent copy of the data? 

 

216. What is immediate vs. eventual consistency? 

• For any given read, how likely is it the data may be stale? 
• Immediate Consistency – reads always return the most recent data 

• Consistency Level ALL guarantees immediate consistency, because all replica nodes are checked and compared before a result is returned 

• Highest latency because all replicas are checked and compared 

• Eventual Consistency – reads may return stale data 

• Consistency Level ONE carries the highest risk of stale data, because the replica from the first node to respond is immediately returned 

• Lowest latency because the first replica is immediately returned 

 

217. What does it mean to tune consistency? 

• Reads and writes may each be set to a specific consistency level 

if (nodes_written + nodes_read) > replication_factor 
then immediate consistency 

 

218. How do you choose a consistency level? 

• In any given scenario, is the value of immediate consistency worth the latency cost? 

• Netflix uses CL ONE and measures its "eventual" consistency in milliseconds 

• Consistency Level ONE is your friend ... 

Consistency Level ONE	Consistency Level QUORUM	Consistency Level ALL
Lowest latency	Higher latency (than ONE)	Highest latency
Highest throughput	Lower throughput	Lowest throughput
Highest availability	Higher availability (than ALL)	Lowest availability
Stale read possible  (if read CL + write CL < RF)	No stale reads  (if read and write at quorum)	No stale reads  (if either read or write at ALL)

• If "stale" is measured in milliseconds, how much are those milliseconds worth? 

 

219. What is the Gossip protocol? 

• Once per second, each node contacts 1 to 3 others, requesting and sharing updates about 

•  Known node states ("heartbeats") 

•  Known node locations 

•  Requests and acknowledgments are timestamped, so information is continually updated and discarded 

 

220. What is write path 

• On a write, Cassandra first appends writes to the commit log on disk 

• The write is durable once the data is in the commit log 
• Actually - once fsync is called and the OS flushes its own cache to disk 

• The commit log is append only, so there is no seek necessary for the append 

• Assuming a dedicated disk for the commit log (a recommended practice) 

• A write also stores the data in memory

• In a structure called the memtable

• A write is successful once written to the commit log and memory 

• Very fast -Very little disk I/O at time of write 

 

221. Details of Memtables and SSTables 

•  Memtables are organized in sorted order by partition key 

•  There is one memtable per table per node 

•  Updates to a column values in the memtable overwrites the existing column values 

•  Accessing them is very fast since they are in-memory 

•  Updates are also merged in-memory for a given partition key 

•  Memtables are eventually flushed to SSTables (Sorted String Tables) on disk 

•  So they don't grow too large in memory 

• Flushed using sequential I/O - no random seeking so it's fast 

• SSTables are immutable once they are written to disk 

•  Updates to data already in an SSTable go into a memtable, then eventually into a different SSTable 

• So data for a given partition key may be in several places 

After flushing, the memtable is emptied. 

 

222. About SSTables 

• SSTable is immutable once it is written
– Mutations to keys already in an SSTable eventually end up in another SSTable – Never any updates to existing data in SSTable
– Meaning no disk seeks on write - speeds up writes 

• Reads may need to go to multiple SSTables for a given key
– Because multiple writes may have created fragments in multiple SSTables for a given key 

• SSTables contain structures to speed up reads – Bloom filters and indexes

223. What is Compaction 

• Compaction: Merges SSTables for a data table into one SSTable - eliminating fragments 

• Reduces number of SSTables to be accessed for a read request 
• Runs asynchronously in the background 
• Uses sequential I/O - fast 
• When merging multiple column values, latest timestamped value is used

 

224. What is data availability & access in Cassandra ?

• Cassandra must access multiple locations to read data 

• Across replicas 
• Data can be replicated across multiple replicas 
• Replicas may not be consistent at any given time (eventual consistency) 

• Within a replica, data may be 

• In an unflushed memtable 
• In multiple SSTables 
• In a cache 

 

225. What is Client read requests 

• Client requests are made to a coordinator 
• The coordinator contacts replicas based on the request CL (1) 

• In the request below, we have RF=3 (data on nodes R1, R2, and R3) 

• Assume CL=QUORUM - so the coordinator contacts two replicas - in this case R1 and R3 

 

226. What is Merging replica data 

•  The coordinator may read data from multiple replicas 
• The data may not be consistent (a write may not have propagated) 

•  If multiple replicas are contacted, the rows from each replica are compared for consistency 

• If consistent, then the data is just returned 
• If not consistent, then the most recent data is used 

• Based on the timestamp value that is contained in the internal storage cell 

•  Cassandra uses a mechanism, read repair, to ensure that all replicas are updated to the latest version of data

 

227. How Read processing in a node 

• To satisfy a read request for a given partition key, a node combines data from 

– Any unflushed memtables
– All SSTables on the node that contain data for that partition key 

 

228. how to Optimizing reads by bloom filters 

• Cassandra uses Bloom filters to minimize SSTable reads

 • Each SSTable read is a disk I/O, so we want to minimize them 

• Bloom filters are used to check if an SSTable has data for a particular partition key 

•  One per SSTable 

•  Saved on disk, but kept in memory (off heap) 

•  On a read, the node checks the Bloom filter for each SSTable 

•  The SSTable is only read from disk if the Bloom filter indicates there is data for the key 

•  This helps make Cassandra very performant on reads

 

229. What is Full read path 

•  Row Cache (off heap): If found here, just return the data, otherwise continue with steps below 

•  Memtable (on heap): Read current memtable, and memtables awaiting flush. Get row fragments for the given row. 

•  Bloom Filter (off heap): Check for each SSTable to build list of candidate SS Tables 

•  Key Cache (on heap): (If enabled) For each SSTable from above, probe the key cache to get position in data file. This may miss. 

•  SSTable Index summary (on heap): Probe here to find start of range in index file, seek to this position in the index file, then scan until you find the key 

•  SSTable (on disk): Seek to the row position in the SSTable and get the data 

•  Merge all row fragments, and reconcile duplicates via timestamp 

•  Update row cache 

•  Return results to client 

 

 

230. Summary of Cassandra terminology 

•  A Cassandra cluster is comprised of peer-to-peer nodes logically organized into racks within data centers 

•  Any node may coordinate any request issued by a Cassandra client 

•  Data is organized into partitions ("rows") identified by tokens in a 2127-1 integer range 

•  The total token range is treated internally as a ring whose segments are owned by nodes 

•  Nodes are identified by the highest token in their segment of the total range 

•  A node's partitioner hashes a token from the partition key of a value being written 

•  The first replica ("copy") of a partition is written to the node owning the primary range containing its token

•  Replication factor (RF) determines how many replicas ("copies") are made of each partition 

•  Replication strategy determines how replicas are distributed across the cluster 

•  A per-request consistency level (CL) determines how many nodes must acknowledge 

•  Nodes continually exchange state and location information via the Gossip protocol 

•  Each node includes a Snitch which tracks and reports on the current cluster topology