• A common language (for the two ends to understand)
• A way to address who you want to communicate with
• A Connection (so the content of the communication can reach the recipients)

A set of protocols that define how two or more devices can communicate with each other.

To learn more about TCP/IP, read here

Ethernet simply refers to the most common type of Local Area Network (LAN) used today. A LAN—in contrast to a WAN (Wide Area Network), which spans a larger geographical area—is a connected network of computers in a small area, like your office, college campus, or even home.

A MAC address is a unique identification number or code used to identify individual devices on the network.

Packets that are sent on the ethernet are always coming from a MAC address and sent to a MAC address. If a network adapter is receiving a packet, it is comparing the packet’s destination MAC address to the adapter’s own MAC address.

When a device sends a packet to the broadcast MAC address (FF:FF:FF:FF:FF:FF​), it is delivered to all stations on the local network. Ethernet broadcasts are used to resolve IP addresses to MAC addresses (by ARP) at the data link layer.

An Internet Protocol address (IP address) is a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication.An IP address serves two main functions: host or network interface identification and location addressing.

A Subnet mask is a 32-bit number that masks an IP address and divides the IP addresses into network addresses and host addresses. Subnet Mask is made by setting network bits to all "1"s and setting host bits to all "0"s. Within a given network, out of the total usable host addresses, two are always reserved for specific purposes and cannot be allocated to any host. These are the first address, which is reserved as a network address (a.k.a network ID), and the last address used for network broadcast.

Example

• Application: user end (HTTP is here)
• Presentation: establishes context between application-layer entities (Encryption is here)
• Session: establishes, manages, and terminates the connections
• Transport: transfers variable-length data sequences from a source to a destination host (TCP & UDP are here)
• Network: transfers datagrams from one network to another (IP is here)
• Data link: provides a link between two directly connected nodes (MAC is here)
• Physical: the electrical and physical spec of the data connection (Bits are here)

You can read more about the OSI model in penguintutor.com

Unicast: One-to-one communication where there is one sender and one receiver.

Broadcast: Sending a message to everyone in the network. The address ff:ff:ff:ff:ff:ff is used for broadcasting. Two common protocols which use broadcast are ARP and DHCP.

Multicast: Sending a message to a group of subscribers. It can be one-to-many or many-to-many.

CSMA/CD stands for Carrier Sense Multiple Access / Collision Detection. Its primary focus is to manage access to a shared medium/bus where only one host can transmit at a given point in time.

CSMA/CD algorithm:

A router, switch, and hub are all network devices used to connect devices in a local area network (LAN). However, each device operates differently and has its specific use cases. Here is a brief description of each device and the differences between them:

Three collision domains and one broadcast domain

A router is a physical or virtual appliance that passes information between two or more packet-switched computer networks. A router inspects a given data packet’s destination Internet Protocol address (IP address), calculates the best way for it to reach its destination, and then forwards it accordingly.

Network Address Translation (NAT) is a process in which one or more local IP addresses are translated into one or more Global IP address and vice versa in order to provide Internet access to the local hosts.

A proxy server acts as a gateway between you and the internet. It’s an intermediary server separating end users from the websites they browse.

If you’re using a proxy server, internet traffic flows through the proxy server on its way to the address you requested. The request then comes back through that same proxy server (there are exceptions to this rule), and then the proxy server forwards the data received from the website to you.

Proxy servers provide varying levels of functionality, security, and privacy depending on your use case, needs, or company policy.

TCP 3-way handshake or three-way handshake is a process that is used in a TCP/IP network to make a connection between server and client.

A three-way handshake is primarily used to create a TCP socket connection. It works when:

From wikipedia: “the length of time it takes for a signal to be sent plus the length of time it takes for an acknowledgment of that signal to be received”

Bonus question: what is the RTT of LAN?

TCP establishes a connection between the client and the server to guarantee the order of the packages, on the other hand, UDP does not establish a connection between the client and server and doesn’t handle package orders. This makes UDP more lightweight than TCP and a perfect candidate for services like streaming.

Penguintutor.com provides a good explanation.

A default gateway serves as an access point or IP router that a networked computer uses to send information to a computer in another network or the internet.

ARP stands for Address Resolution Protocol. When you try to ping an IP address on your local network, say 192.168.1.1, your system has to turn the IP address 192.168.1.1 into a MAC address. This involves using ARP to resolve the address, hence its name.

Systems keep an ARP look-up table where they store information about what IP addresses are associated with what MAC addresses. When trying to send a packet to an IP address, the system will first consult this table to see if it already knows the MAC address. If there is a value cached, ARP is not used.

It stands for Dynamic Host Configuration Protocol and allocates IP addresses, subnet masks, and gateways to hosts. This is how it works:

• A host upon entering a network broadcasts a message in search of a DHCP server (DHCP DISCOVER)
• An offer message is sent back by the DHCP server as a packet containing lease time, subnet mask, IP addresses, etc (DHCP OFFER)
• Depending on which offer is accepted, the client sends back a reply broadcast letting all DHCP servers know (DHCP REQUEST)
• The server sends an acknowledgment (DHCP ACK)

Read more here

It is possible to have two DHCP servers on the same network, however, it is not recommended, and it is important to configure them carefully to prevent conflicts and configuration problems.

• SSL (Secure Sockets Layer) tunneling is a technique used to establish a secure, encrypted connection between two endpoints over an insecure network, such as the Internet. The SSL tunnel is created by encapsulating the traffic within an SSL connection, which provides confidentiality, integrity, and authentication.

Here’s how SSL tunneling works:

• IPv6 (Internet Protocol version 6) is the latest version of the Internet Protocol (IP), which is used to identify and communicate with devices on a network. IPv6 addresses are 128-bit addresses and are expressed in hexadecimal notation, such as 2001:0db8:85a3:0000:0000:8a2e:0370:7334.

There are several reasons why we should consider using IPv6 over IPv4:

MTU stands for Maximum Transmission Unit. It’s the size of the largest PDU (protocol Data Unit) that can be sent in a single transaction.

With the IPv4 protocol, the router can fragment the PDU and then send all the fragmented PDU through the transaction.

With IPv6 protocol, it issues an error to the user’s computer.

False. Ping is actually using ICMP (Internet Control Message Protocol) which is a network protocol used to send diagnostic messages and control messages related to network communication.

• ICMP stands for Internet Control Message Protocol. It is a protocol used for diagnostic and control purposes in IP networks. It is a part of the Internet Protocol suite, operating at the network layer.

ICMP messages are used for a variety of purposes, including: 1. Error reporting: ICMP messages are used to report errors that occur in the network, such as a packet that could not be delivered to its destination. 2. Ping: ICMP is used to send ping messages, which are used to test whether a host or network is reachable and to measure the round-trip time for packets. 3. Path MTU discovery: ICMP is used to discover the Maximum Transmission Unit (MTU) of a path, which is the largest packet size that can be transmitted without fragmentation. 4. Traceroute: ICMP is used by the traceroute utility to trace the path that packets take through the network. 5. Router discovery: ICMP is used to discover the routers in a network.

NAT stands for Network Address Translation. It’s a way to map multiple local private addresses to a public one before transferring the information. Organizations that want multiple devices to employ a single IP address use NAT, as do most home routers. For example, your computer’s private IP could be 192.168.1.100, but your router maps the traffic to its public IP (e.g. 1.1.1.1). Any device on the internet would see the traffic coming from your public IP (1.1.1.1) instead of your private IP (192.168.1.100).

Several factors can affect network performance, including:

APIPA is a set of IP addresses that devices are allocated when the main DHCP server is not reachable

APIPA uses the IP range: 169.254.0.1 - 169.254.255.254.

The control plane is a part of the network that decides how to route and forward packets to a different location.

The data plane is a part of the network that actually forwards the data/packets.

It refers to monitoring and management functions.

Control Plane.

OSPF (Open Shortest Path First) is a routing protocol that can be implemented on various types of routers. In general, OSPF is supported on most modern routers, including those from vendors such as Cisco, Juniper, and Huawei. The protocol is designed to work with IP-based networks, including both IPv4 and IPv6. Also, it uses a hierarchical network design, where routers are grouped into areas, with each area having its own topology map and routing table. This design helps to reduce the amount of routing information that needs to be exchanged between routers and improve network scalability.

The OSPF 4 Types of routers are:

• Internal Router
• Area Border Routers
• Autonomous Systems Boundary Routers
• Backbone Routers

Learn more about OSPF router types: https://www.educba.com/ospf-router-types/

Latency is the time taken for information to reach its destination from the source.

Bandwidth is the capacity of a communication channel to measure how much data the latter can handle over a specific time period. More bandwidth would imply more traffic handling and thus more data transfer.

Throughput refers to the measurement of the real amount of data transferred over a certain period of time across any transmission channel.

Latency. To have good latency, a search query should be forwarded to the closest data center.

Throughput. To have good throughput, the upload stream should be routed to an underutilized link.

Within a Spine & Leaf network there are two main tipology of switches:

• Spine Switches: Spine switches are high-performance switches arranged in a spine layer. These switches act as the core of the network and are typically interconnected with each leaf switch. Each spine switch is connected to all the leaf switches in the data center.
• Leaf Switches: Leaf switches are connected to end devices like servers, storage arrays, and other networking equipment. Each leaf switch is connected to every spine switch in the data center. This creates a non-blocking, full-mesh connectivity between leaf and spine switches, ensuring any leaf switch can communicate with any other leaf switch with maximum throughput.

The Spine & Leaf architecture has become increasingly popular in data centers due to its ability to handle the demands of modern cloud computing, virtualization, and big data applications, providing a scalable, high-performance, and reliable network infrastructure

Network congestion occurs when there is too much data to transmit on a network and it doesn’t have enough capacity to handle the demand.
This can lead to increased latency and packet loss. The causes can be multiple, such as high network usage, large file transfers, malware, hardware issues, or network design problems.
To prevent network congestion, it’s important to monitor your network usage and implement strategies to limit or manage the demand.

00110011110100011101

Read more [here](https://www.globalsign.com/en/blog/what-is-hsts-and-how-do-i-use-it#:~:text=HTTP%20Strict%20Transport%20Security%20(HSTS,and%20back%20to%20the%20browser.)

The internet refers to a network of networks, transferring huge amounts of data around the globe.
The World Wide Web is an application running on millions of servers, on top of the internet, accessed through what is known as the web browser

ISP (Internet Service Provider) is the local internet company provider.

From the book “Operating Systems: Three Easy Pieces”:

“responsible for making it easy to run programs (even allowing you to seemingly run many at the same time), allowing programs to share memory, enabling programs to interact with devices, and other fun stuff like that”.

A process is a running program. A program is one or more instructions and the program (or process) is executed by the operating system.

It would support the following:

False. It was true in the past but today’s operating systems perform lazy loading, which means only the relevant pieces required for the process to run are loaded first.

Inter-process communication (IPC) refers to the mechanisms provided by an operating system that allow processes to manage shared data.

Even when using a system with one physical CPU, it’s possible to allow multiple users to work on it and run programs. This is possible with time sharing, where computing resources are shared in a way it seems to the user, the system has multiple CPUs, but in fact it’s simply one CPU shared by applying multiprogramming and multi-tasking.

Somewhat the opposite of time sharing. While in time sharing a resource is used for a while by one entity and then the same resource can be used by another resource, in space sharing the space is shared by multiple entities but in a way where it’s not being transferred between them.
It’s used by one entity, until this entity decides to get rid of it. Take for example storage. In storage, a file is yours, until you decide to delete it.

CPU scheduler

Virtual memory combines your computer’s RAM with temporary space on your hard disk. When RAM runs low, virtual memory helps to move data from RAM to a space called a paging file. Moving data to paging file can free up the RAM, so your computer can complete its work. In general, the more RAM your computer has, the faster the programs run. https://www.minitool.com/lib/virtual-memory.html

Demand paging is a memory management technique where pages are loaded into physical memory only when accessed by a process. It optimizes memory usage by loading pages on demand, reducing startup latency and space overhead. However, it introduces some latency when accessing pages for the first time. Overall, it’s a cost-effective approach for managing memory resources in operating systems.

The idea: 1. If resources are shared between 2 or more entities (for example shared memory segments between 2 processes), the resources don’t need to be copied for every entity, but rather every entity has a READ operation access permission on the shared resource. (the shared segments are marked as read-only) (Think of every entity having a pointer to the location of the shared resource, which can be dereferenced to read its value) 2. If one entity would perform a WRITE operation on a shared resource, a problem would arise, since the resource also would be permanently changed for ALL other entities sharing it. (Think of a process modifying some variables on the stack, or allocatingy some data dynamically on the heap, these changes to the shared resource would also apply for ALL other processes, this is definitely an undesirable behaviour) 3. As a solution only, if a WRITE operation is about to be performed on a shared resource, this resource gets COPIED first and then the changes are applied.

The kernel is part of the operating system and is responsible for tasks like:

True

POSIX (Portable Operating System Interface) is a set of standards that define the interface between a Unix-like operating system and application programs.

A semaphore is a synchronization primitive used in operating systems and concurrent programming to control access to shared resources. It’s a variable or abstract data type that acts as a counter or a signaling mechanism for managing access to resources by multiple processes or threads.

Cache: Cache is usually used when processes are reading and writing to the disk to make the process faster, by making similar data used by different programs easily accessible. Buffer: Reserved place in RAM, which is used to hold data for temporary purposes.

Virtualization uses software to create an abstraction layer over computer hardware, that allows the hardware elements of a single computer - processors, memory, storage and more - to be divided into multiple virtual computers, commonly called virtual machines (VMs).

Red Hat: “A hypervisor is software that creates and runs virtual machines (VMs). A hypervisor, sometimes called a virtual machine monitor (VMM), isolates the hypervisor operating system and resources from the virtual machines and enables the creation and management of those VMs.”

Read more here

Hosted hypervisors and bare-metal hypervisors.

Due to having its own drivers and a direct access to hardware components, a baremetal hypervisor will often have better performances along with stability and scalability.

On the other hand, there will probably be some limitation regarding loading (any) drivers so a hosted hypervisor will usually benefit from having a better hardware compatibility.

Operating system virtualization Network functions virtualization Desktop virtualization

Yes, it’s a operating-system-level virtualization, where the kernel is shared and allows to use multiple isolated user-spaces instances.

The introduction of virtual machines allowed companies to deploy multiple business applications on the same hardware, while each application is separated from each other in secured way, where each is running on its own separate operating system.

Yes, virtual machines are still relevant even in the age of containers. While containers provide a lightweight and portable alternative to virtual machines, they do have certain limitations. Virtual machines still matter because they offer isolation and security, can run different operating systems, and are good for legacy apps. Containers limitations for example are sharing the host kernel.

Prometheus is a popular open-source systems monitoring and alerting toolkit, originally developed at SoundCloud. It is designed to collect and store time-series data, and to allow for querying and analysis of that data using a powerful query language called PromQL. Prometheus is frequently used to monitor cloud-native applications, microservices, and other modern infrastructure.

Some of the main features of Prometheus include:

1. Data model: Prometheus uses a flexible data model that allows users to organize and label their time-series data in a way that makes sense for their particular use case. Labels are used to identify different dimensions of the data, such as the source of the data or the environment in which it was collected.

2. Pull-based architecture: Prometheus uses a pull-based model to collect data from targets, meaning that the Prometheus server actively queries its targets for metrics data at regular intervals. This architecture is more scalable and reliable than a push-based model, which would require every target to push data to the server.

3. Time-series database: Prometheus stores all of its data in a time-series database, which allows users to perform queries over time ranges and to aggregate and analyze their data in various ways. The database is optimized for write-heavy workloads, and can handle a high volume of data with low latency.

4. Alerting: Prometheus includes a powerful alerting system that allows users to define rules based on their metrics data and to send alerts when certain conditions are met. Alerts can be sent via email, chat, or other channels, and can be customized to include specific details about the problem.

5. Visualization: Prometheus has a built-in graphing and visualization tool, called PromDash, which allows users to create custom dashboards to monitor their systems and applications. PromDash supports a variety of graph types and visualization options, and can be customized using CSS and JavaScript.

Overall, Prometheus is a powerful and flexible tool for monitoring and analyzing systems and applications, and is widely used in the industry for cloud-native monitoring and observability.

From Prometheus documentation: “if you need 100% accuracy, such as for per-request billing”.

The Prometheus architecture consists of four major components:

1. Prometheus Server: The Prometheus server is responsible for collecting and storing metrics data. It has a simple built-in storage layer that allows it to store time-series data in a time-ordered database.

2. Client Libraries: Prometheus provides a range of client libraries that enable applications to expose their metrics data in a format that can be ingested by the Prometheus server. These libraries are available for a range of programming languages, including Java, Python, and Go.

3. Exporters: Exporters are software components that expose existing metrics from third-party systems and make them available for ingestion by the Prometheus server. Prometheus provides exporters for a range of popular technologies, including MySQL, PostgreSQL, and Apache.

4. Alertmanager: The Alertmanager component is responsible for processing alerts generated by the Prometheus server. It can handle alerts from multiple sources and provides a range of features for deduplicating, grouping, and routing alerts to appropriate channels.

Overall, the Prometheus architecture is designed to be highly scalable and resilient. The server and client libraries can be deployed in a distributed fashion to support monitoring across large-scale, highly dynamic environments

Compared to other monitoring solutions, such as InfluxDB, Prometheus is known for its high performance and scalability. It can handle large volumes of data and can easily be integrated with other tools in the monitoring ecosystem. InfluxDB, on the other hand, is known for its ease of use and simplicity. It has a user-friendly interface and provides easy-to-use APIs for collecting and querying data.

Another popular solution, Nagios, is a more traditional monitoring system that relies on a push-based model for collecting data. Nagios has been around for a long time and is known for its stability and reliability. However, compared to Prometheus, Nagios lacks some of the more advanced features, such as multi-dimensional data model and powerful query language.

Overall, the choice of a monitoring solution depends on the specific needs and requirements of the organization. While Prometheus is a great choice for large-scale monitoring and alerting, InfluxDB may be a better fit for smaller environments that require ease of use and simplicity. Nagios remains a solid choice for organizations that prioritize stability and reliability over advanced features.

In Prometheus, an instance refers to a single target that is being monitored. For example, a single server or service. A job is a set of instances that perform the same function, such as a set of web servers serving the same application. Jobs allow you to define and manage a group of targets together.

In essence, an instance is an individual target that Prometheus collects metrics from, while a job is a collection of similar instances that can be managed as a group.

1. Counter: A monotonically increasing value used for tracking counts of events or samples. Examples include the number of requests processed or the total number of errors encountered.

2. Gauge: A value that can go up or down, such as CPU usage or memory usage. Unlike counters, gauge values can be arbitrary, meaning they can go up and down based on changes in the system being monitored.

3. Histogram: A set of observations or events that are divided into buckets based on their value. Histograms help in analyzing the distribution of a metric, such as request latencies or response sizes.

4. Summary: A summary is similar to a histogram, but instead of buckets, it provides a set of quantiles for the observed values. Summaries are useful for monitoring the distribution of request latencies or response sizes over time.

Prometheus also supports various functions and operators for aggregating and manipulating metrics, such as sum, max, min, and rate. These features make it a powerful tool for monitoring and alerting on system metrics.

The exporter acts as a server, listening on a specific network port for requests from Prometheus to scrape metrics. It collects metrics from the third-party system or application and transforms them into a format that can be understood by Prometheus. The exporter then exposes these metrics to Prometheus via an HTTP endpoint, making them available for collection and analysis.

Exporters are commonly used to monitor various types of infrastructure components such as databases, web servers, and storage systems. For example, there are exporters available for monitoring popular databases such as MySQL and PostgreSQL, as well as web servers like Apache and Nginx.

Overall, exporters are a critical component of the Prometheus ecosystem, allowing for the monitoring of a wide range of systems and applications, and providing a high degree of flexibility and extensibility to the platform.

1. Label carefully: Careful and consistent labeling of metrics is crucial for effective querying and alerting. Labels should be clear, concise, and include all relevant information about the metric.

2. Keep metrics simple: The metrics exposed by exporters should be simple and focus on a single aspect of the system being monitored. This helps avoid confusion and ensures that the metrics are easily understandable by all members of the team.

3. Use alerting sparingly: While alerting is a powerful feature of Prometheus, it should be used sparingly and only for the most critical issues. Setting up too many alerts can lead to alert fatigue and result in important alerts being ignored. It is recommended to set up only the most important alerts and adjust the thresholds over time based on the actual frequency of alerts.

sum(rate(http_requests_total[1h]))

In this query, http_requests_total is the name of the metric that tracks the total number of HTTP requests, and the rate function calculates the per-second rate of requests over the last hour. The sum function then adds up all of the requests to give you the total number of requests in the last hour.

You can adjust the time range by changing the duration in the rate function. For example, if you wanted to get the total number of requests in the last day, you could change the function to rate(http_requests_total[1d]).

HA stands for High Availability. This means that the system is designed to be highly reliable and always available, even in the face of failures or other issues. In practice, this typically involves setting up multiple instances of Prometheus and ensuring that they are all synchronized and able to work together seamlessly. This can be achieved through a variety of techniques, such as load balancing, replication, and failover mechanisms. By implementing HA in Prometheus, users can ensure that their monitoring data is always available and up-to-date, even in the face of hardware or software failures, network issues, or other problems that might otherwise cause downtime or data loss.

Here’s an example of how to join two metrics using the join() function:

sum_series(
  join(
    on(service, instance) request_count_total,
    on(service, instance) error_count_total,
  )
)

In this example, the join() function combines the request_count_total and error_count_total time series based on their service and instance label values. The sum_series() function then calculates the sum of the resulting time series

For example, if you have a metric called http_requests_total with a label called method, and you want to return all the values of the method label, you can use the following query:

label_values(http_requests_total, method)

This will return a list of all the values for the method label in the http_requests_total metric. You can then use this list in further queries or to filter your data.

100 * sum(rate(process_cpu_user_seconds_total{job="<job-name>"}[<time-period>])) by (instance) / (<time-period> * <num-cpu-cores>)

Here, is the name of the job you want to query, is the time range you want to query (e.g. 5m, 1h), and is the number of CPU cores on the machine you are querying.

For example, to get the CPU usage in percentage for the last 5 minutes for a job named my-job running on a machine with 4 CPU cores, you can use the following query:

100 * sum(rate(process_cpu_user_seconds_total{job="my-job"}[5m])) by (instance) / (5m * 4)

• Strong and static typing - the type of the variables can’t be changed over time and they have to be defined at compile time
• Simplicity
• Fast compile times
• Built-in concurrency
• Garbage collected
• Platform independent
• Compile to standalone binary - anything you need to run your app will be compiled into one binary. Very useful for version management in run-time.

Go also has good community.

The result is the same, a variable with the value 2.

With var x int = 2 we are setting the variable type to integer while with x := 2 we are letting Go figure out by itself the type.

This should be answered based on your usage but some examples are:

It looks what unicode value is set at 101 and uses it for converting the integer to a string. If you want to get “101” you should use the package “strconv” and replace y = string(x) with y = strconv.Itoa(x)

Constants in Go can only be declared using constant expressions. But x, y and their sum is variable.
const initializer x + y is not a constant

Go’s iota identifier is used in const declarations to simplify definitions of incrementing numbers. Because it can be used in expressions, it provides a generality beyond that of simple enumerations.
x and y in the first iota group, z in the second.
Iota page in Go Wiki

It avoids having to declare all the variables for the returns values. It is called the blank identifier.
answer in SO

Since the first iota is declared with the value 3 (+ 3), the next one has the value 4

Output: 2 1 3

Aritcle about sync/waitgroup

Golang package sync

Output:
1 [5 5 5 5]
1 [5 5 5 5]
2 [5 5 5 5 5]
2 [1 2 3 4]


In mod1 a is link, and when we’re using a[i], we’re changing s1 value to. But in mod2, append creates new slice, and we’re changing only a value, not s2.

Aritcle about arrays, Blog post about append

Output: 3

Golang container/heap package

MongoDB advantages are as following:

• Schemaless
• Easy to scale-out
• No complex joins
• Structure of a single object is clear

The main difference is that SQL databases are structured (data is stored in the form of tables with rows and columns - like an excel spreadsheet table) while NoSQL is unstructured, and the data storage can vary depending on how the NoSQL DB is set up, such as key-value pair, document-oriented, etc.

SQL (Structured Query Language) is a standard language for relational databases (like MySQL, MariaDB, …).
It’s used for reading, updating, removing and creating data in a relational database.

The main difference is that SQL databases are structured (data is stored in the form of tables with rows and columns - like an excel spreadsheet table) while NoSQL is unstructured, and the data storage can vary depending on how the NoSQL DB is set up, such as key-value pair, document-oriented, etc.

SQL - Best used when data integrity is crucial. SQL is typically implemented with many businesses and areas within the finance field due to it’s ACID compliance.

NoSQL - Great if you need to scale things quickly. NoSQL was designed with web applications in mind, so it works great if you need to quickly spread the same information around to multiple servers

Additionally, since NoSQL does not adhere to the strict table with columns and rows structure that Relational Databases require, you can store different data types together.

Select *
From Customers;

Select Items_in_cart
From Customers
Where Customer_Name = “John Smith”;

Select SUM(Cash_spent_to_Date) as SUM_CASH
From Customers;

Select count(1) as Number_of_People_w_items
From Customers
where Items_in_cart > 0;

You would join them on the unique key. In this case, the unique key is Customer_ID in both the Customers table and Orders table

Select c.Customer_Name, o.Item
From Customers c
Left Join Orders o
On c.Customer_ID = o.Customer_ID;

with cat_food as (
Select Customer_ID, SUM(Price) as TOTAL_PRICE
From Orders
Where Item like “%Cat Food%”
Group by Customer_ID
)
Select Customer_name, TOTAL_PRICE
From Customers c
Inner JOIN cat_food f
ON c.Customer_ID = f.Customer_ID
where c.Customer_ID in (Select Customer_ID from cat_food);

Although this was a simple statement, the “with” clause really shines when a complex query needs to be run on a table before joining to another. With statements are nice, because you create a pseudo temp when running your query, instead of creating a whole new table.

The Sum of all the purchases of cat food weren’t readily available, so we used a with statement to create the pseudo table to retrieve the sum of the prices spent by each customer, then join the table normally.

SELECT count(*)
FROM shawarma_purchases
WHERE
  purchased_at >= '2017-01-01' AND
  purchased_at <= '2017-31-12'

When you use a function (YEAR(purchased_at)) it has to scan the whole database as opposed to using indexes and basically the column as it is, in its natural state.

You can read about TripleO right here

There are many reasons for that. One for example: you can’t remove router if there are active ports assigned to it.

Not by default. Object Storage API limits the maximum to 5GB per object but it can be adjusted.

False. Two objects can have the same name if they are in different containers.

Using:

A list of services and their endpoints

The Elastic Stack consists of:

• Elasticsearch
• Kibana
• Logstash
• Beats
• Elastic Hadoop
• APM Server

Elasticsearch, Logstash and Kibana are also known as the ELK stack.

From the official docs:

“Elasticsearch is a distributed document store. Instead of storing information as rows of columnar data, Elasticsearch stores complex data structures that have been serialized as JSON documents”

From the blog:

“Logstash is a powerful, flexible pipeline that collects, enriches and transports data. It works as an extract, transform & load (ETL) tool for collecting log messages.”

Beats are lightweight data shippers. These data shippers installed on the client where the data resides. Examples of beats: Filebeat, Metricbeat, Auditbeat. There are much more.

From the official docs:

“Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.”

The process may vary based on the chosen architecture and the processing you may want to apply to the logs. One possible workflow is:

This is where data is stored and also where different processing takes place (e.g. when you search for a data).

Part of a master node responsibilities:

• Track the status of all the nodes in the cluster
• Verify replicas are working and the data is available from every data node.
• No hot nodes (no data node that works much harder than other nodes)

While there can be multiple master nodes in reality only of them is the elected master node.

A node which responsible for processing the data according to ingest pipeline. In case you don’t need to use logstash then this node can receive data from beats and process it, similarly to how it can be processed in Logstash.

From the official docs:

Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes. They join the cluster and receive the full cluster state, like every other node, and they use the cluster state to route requests directly to the appropriate place(s).

Index in Elasticsearch is in most cases compared to a whole database from the SQL/NoSQL world.
You can choose to have one index to hold all the data of your app or have multiple indices where each index holds different type of your app (e.g. index for each service your app is running).

The official docs also offer a great explanation (in general, it’s really good documentation, as every project should have):

“An index can be thought of as an optimized collection of documents and each document is a collection of fields, which are the key-value pairs that contain your data”

An index is split into shards and documents are hashed to a particular shard. Each shard may be on a different node in a cluster and each one of the shards is a self contained index.
This allows Elasticsearch to scale to an entire cluster of servers.

From the official docs:

“An inverted index lists every unique word that appears in any document and identifies all of the documents each word occurs in.”

Continuing with the comparison to SQL/NoSQL a Document in Elasticsearch is a row in table in the case of SQL or a document in a collection in the case of NoSQL. As in NoSQL a document is a JSON object which holds data on a unit in your app. What is this unit depends on the your app. If your app related to book then each document describes a book. If you are app is about shirts then each document is a shirt.

Red means some data is unavailable in your cluster. Some shards of your indices are unassigned. There are some other states for the cluster. Yellow means that you have unassigned shards in the cluster. You can be in this state if you have single node and your indices have replicas. Green means that all shards in the cluster are assigned to nodes and your cluster is healthy.

False. From the official docs:

“Each indexed field has a dedicated, optimized data structure. For example, text fields are stored in inverted indices, and numeric and geo fields are stored in BKD trees.”

In a network/cloud environment where failures can be expected any time, it is very useful and highly recommended to have a failover mechanism in case a shard/node somehow goes offline or disappears for whatever reason. To this end, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.

Term Frequency is how often a term appears in a given document and Document Frequency is how often a term appears in all documents. They both are used for determining the relevance of a term by calculating Term Frequency / Document Frequency.

“The index is actively being written to”. More about the phases here

It creates customer index if it doesn’t exists and adds a new document with the field name which is set to “John Dow”. Also, if it’s the first document it will get the ID 1.

Bulk API is used when you need to index multiple documents. For high number of documents it would be significantly faster to use rather than individual requests since there are less network roundtrips.

From the official docs:

“In the query context, a query clause answers the question “How well does this document match this query clause?” Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the _score meta-field.”

“In a filter context, a query clause answers the question “Does this document match this query clause?” The answer is a simple Yes or No — no scores are calculated. Filter context is mostly used for filtering structured data”

There are several possible answers for this question. One of them is as follows:

A small-scale architecture of elastic will consist of the elastic stack as it is. This means we will have beats, logstash, elastcsearch and kibana.
A production environment with large amounts of data can include some kind of buffering component (e.g. Reddis or RabbitMQ) and also security component such as Nginx.

A logstash plugin which modifies information in one format and immerse it in another.

The raw data as it is stored in the index. You can search and filter it.

Total number of documents matching the search results. If not query used then simply the total number of documents.

“Visualize” is where you can create visual representations for your data (pie charts, graphs, …)

Filebeat is used to monitor the logging directories inside of VMs or mounted as a sidecar if exporting logs from containers, and then forward these logs onward for further processing, usually to logstash.

Filebeat is a typical component of the ELK stack, since it was developed by Elastic to work with the other products (Logstash and Kibana). It’s possible to send logs directly to logstash, though this often requires coding changes for the application. Particularly for legacy applications with little test coverage, it might be a better option to use filebeat, since you don’t need to make any changes to the application code.

Read here

False. One harvester harvests one file.

These are pre-configured modules for specific types of logging locations (eg, Traefik, Fargate, HAProxy) to make it easy to configure forwarding logs using filebeat. They have different configurations based on where you’re collecting logs from.

You can generate certificates with the provided elastic utils and change configuration to enable security using certificates model.

According to Martin Kleppmann:

“Many processes running on many machines…only message-passing via an unreliable network with variable delays, and the system may suffer from partial failures, unreliable clocks, and process pauses.”

Another definition: “Systems that are physically separated, but logically connected”

According to the CAP theorem, it’s not possible for a distributed data store to provide more than two of the following at the same time:

Ways to improve:

It’s an architecture in which data is and retrieved from a single, non-shared, source usually exclusively connected to one node as opposed to architectures where the request can get to one of many nodes and the data will be retrieved from one shared location (storage, memory, …).

1. The browser searches for the record of the domain name IP address in the DNS in the following order:
   • Browser cache
   • Operating system cache
   • The DNS server configured on the user’s system (can be ISP DNS, public DNS, …)
2. If it couldn’t find a DNS record locally, a full DNS resolution is started.
3. It connects to the server using the TCP protocol
4. The browser sends an HTTP request to the server
5. The server sends an HTTP response back to the browser
6. The browser renders the response (e.g. HTML)
7. The browser then sends subsequent requests as needed to the server to get the embedded links, javascript, images in the HTML and then steps 3 to 5 are repeated.

TODO: add more details!

I like this definition from blog.christianposta.com:

“An explicitly and purposefully defined interface designed to be invoked over a network that enables software developers to get programmatic access to data and functionality within an organization in a controlled and comfortable way.”

From swagger.io:

“An API specification provides a broad understanding of how an API behaves and how the API links with other APIs. It explains how the API functions and the results to expect when using the API”

False. From swagger.io:

“An API definition is similar to an API specification in that it provides an understanding of how an API is organized and how the API functions. But the API definition is aimed at machine consumption instead of human consumption of APIs.”

An API gateway is like the gatekeeper that controls how different parts talk to each other and how information is exchanged between them.

The API gateway provides a single point of entry for all clients, and it can perform several tasks, including routing requests to the appropriate backend service, load balancing, security and authentication, rate limiting, caching, and monitoring.

By using an API gateway, organizations can simplify the management of their APIs, ensure consistent security and governance, and improve the performance and scalability of their backend services. They are also commonly used in microservices architectures, where there are many small, independent services that need to be accessed by different clients.

Advantages:

• Simplifies API management: Provides a single entry point for all requests, which simplifies the management and monitoring of multiple APIs.
• Improves security: Able to implement security features like authentication, authorization, and encryption to protect the backend services from unauthorized access.
• Enhances scalability: Can handle traffic spikes and distribute requests to backend services in a way that maximizes resource utilization and improves overall system performance.
• Enables service composition: Can combine different backend services into a single API, providing more granular control over the services that clients can access.
• Facilitates integration with external systems: Can be used to expose internal services to external partners or customers, making it easier to integrate with external systems and enabling new business models.

Automation is the act of automating tasks to reduce human intervention or interaction in regards to IT technology and systems.
While automation focuses on a task level, Orchestration is the process of automating processes and/or workflows which consists of multiple tasks that usually across multiple systems.

Data about data. Basically, it describes the type of information that an underlying data will hold.

I can’t answer this for you :)

Domain Specific Language (DSLs) are used to create a customised language that represents the domain such that domain experts can easily interpret it.

Data serialization language used by many technologies today like Kubernetes, Ansible, etc.

True. Because YAML is superset of JSON.

someMultiLineString: |
  look mama
  I can write a multi-line string
  I love YAML

It’s good for use cases like writing a shell script where each line of the script is a different command.

using > will make the multi-line string to fold into a single line

someMultiLineString: >
  This is actually
  a single line
  do not let appearances fool you

They allow you reference values instead of directly writing them and it is used like this:

username: {{ my.user_name }}

Using this: --- For Examples:

document_number: 1
---
document_number: 2

Wikipedia: “In computing, firmware is a specific class of computer software that provides the low-level control for a device’s specific hardware. Firmware, such as the BIOS of a personal computer, may contain basic functions of a device, and may provide hardware abstraction services to higher-level software such as operating systems.”

Avinetworks: HTTP stands for Hypertext Transfer Protocol. HTTP uses TCP port 80 to enable internet communication. It is part of the Application Layer (L7) in OSI Model.

False. It doesn’t maintain state for incoming request.

It consists of:

HTTPS is a secure version of the HTTP protocol used to transfer data between a web browser and a web server. It encrypts the communication using SSL/TLS encryption to ensure that the data is private and secure.

Learn more: https://www.cloudflare.com/learning/ssl/why-is-http-not-secure/

HTTP is stateless. To share state, we can use Cookies.

TODO: explain what is actually a Cookie

The server didn’t receive a response from another server it communicates with in a timely manner.

A proxy is a server that acts as a middleman between a client device and a destination server. It can help improve privacy, security, and performance by hiding the client’s IP address, filtering content, and caching frequently accessed data.

A reverse proxy is a type of proxy server that sits between a client and a server, but it is used to manage traffic going in the opposite direction of a traditional forward proxy. In a forward proxy, the client sends requests to the proxy server, which then forwards them to the destination server. However, in a reverse proxy, the client sends requests to the destination server, but the requests are intercepted by the reverse proxy before they reach the server.

Wikipedia: “The X-Forwarded-For (XFF) HTTP header field is a common method for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.”

A load balancer accepts (or denies) incoming network traffic from a client, and based on some criteria (application related, network, etc.) it distributes those communications out to servers (at least one).

L4 and L7

Yes, you can use DNS for performing load balancing.

Recommended read:

• Red Hat Article

Cons:

You would like to make sure the user doesn’t lose the current session data.

Cookies. There are application based cookies and duration based cookies.

The maximum timeout value can be set between 1 and 3,600 seconds on both GCP and AWS.

The Creative Commons license is a set of copyright licenses that allow creators to share their work with the public while retaining some control over how it can be used. The license was developed as a response to the restrictive standards of traditional copyright laws, which limited access of creative works. Its creators to choose the terms under which their works can be shared, distributed, and used by others. They’re six main types of Creative Commons licenses, each with different levels of restrictions and permissions, the six licenses are:

• Attribution (CC BY): Allows others to distribute, remix, and build upon the work, even commercially, as long as they credit the original creator.
• Attribution-ShareAlike (CC BY-SA): Allows others to remix and build upon the work, even commercially, as long as they credit the original creator and release any new creations under the same license.
• Attribution-NoDerivs (CC BY-ND): Allows others to distribute the work, even commercially, but they cannot remix or change it in any way and must credit the original creator.
• Attribution-NonCommercial (CC BY-NC): Allows others to remix and build upon the work, but they cannot use it commercially and must credit the original creator.
• Attribution-NonCommercial-ShareAlike (CC BY-NC-SA): Allows others to remix and build upon the work, but they cannot use it commercially, must credit the original creator, and must release any new creations under the same license.
• Attribution-NonCommercial-NoDerivs (CC BY-NC-ND): Allows others to download and share the work, but they cannot use it commercially, remix or change it in any way, and must credit the original creator.

Simply stated, the Creative Commons licenses are a way for creators to share their work with the public while retaining some control over how it can be used. The licenses promote creativity, innovation, and collaboration, while also respecting the rights of creators while still encouraging the responsible use of creative works.

More information: https://creativecommons.org/licenses/

In Copyleft, any derivative work must use the same licensing while in permissive licensing there are no such condition. GPL-3 is an example of copyleft license while BSD is an example of permissive license.

CPU cache. Source

A memory leak is a programming error that occurs when a program fails to release memory that is no longer needed, causing the program to consume increasing amounts of memory over time.

The leaks can lead to a variety of problems, including system crashes, performance degradation, and instability. Usually occurring after failed maintenance on older systems and compatibility with new components over time.

SSH HTTP DHCP DNS …

https://idiallo.com/blog/c10k-2016

Pros:

Pros:

Local filesystem Dropbox Google Drive

A file system is a way for computers and other electronic devices to organize and store data files. It provides a structure that helps to organize data into files and directories, making it easier to find and manage information. A file system is crucial for providing a way to store and manage data in an organized manner.

Commonly used filed systems: Windows:

• NTFS
• exFAT

Mac OS:

• HFS+ *APFS

Be careful when asking this question - all companies, regardless of size, have some level of tech debt. Phrase the question in the light that all companies have the deal with this, but you want to see the current pain points they are dealing with

This is a great way to figure how managers deal with unplanned work, and how good they are at setting expectations with projects.

This can give you insights in some of the cool projects a company is working on, and if you would enjoy working on projects like these. This is also a good way to see if the managers are allowing employees to learn and grow with projects outside of the normal work you’d do.

Similar to the tech debt question, this helps you identify any pain points with the company. Additionally, it can be a great way to show how you’d be an asset to the team.

For Example, if they mention they have problem X, and you’ve solved that in the past, you can show how you’d be able to mitigate that problem.

Not only this will tell you what is expected from you, it will also provide big hint on the type of work you are going to do in the first months of your job.

Unit test are a software testing technique that involves systimatically breaking down a system and testing each individual part of the assembly. These tests are automated and can be run repeatedly to allow developers to catch edge case scenarios or bugs quickly while developing.

The main objective of unit tests are to verify each function is producing proper outputs given a set of inputs.

CDN (Content Delivery Network) responsible for distributing content geographically. Part of it, is what is known as edge locations, aka cache proxies, that allows users to get their content quickly due to cache features and geographical distribution.

In single CDN, the whole content is originated from content delivery network.
In multi-CDN, content is distributed across multiple different CDNs, each might be on a completely different provider/cloud.

In a Multi-repo setup, each component is stored in it’s own separate repository. Each repository has it’s own version control history. PROS (Multi-repo):

The ability easily grow in size and capacity based on demand and usage.

The ability to grow but also to reduce based on what is required

Disaster recovery is the process of restoring critical business systems and data after a disruptive event. The goal is to minimize the impact and resume normal business activities quickly. This involves creating a plan, testing it, backing up critical data, and storing it in safe locations. In case of a disaster, the plan is then executed, backups are restored, and systems are hopefully brought back online. The recovery process may take hours or days depending on the damages of infrastructure. This makes business planning important, as a well-designed and tested disaster recovery plan can minimize the impact of a disaster and keep operations going.

Fault Tolerance - The ability to self-heal and return to normal capacity. Also the ability to withstand a failure and remain functional.

High Availability - Being able to access a resource (in some use cases, using different platforms)

wintellect.com: “High availability, simply put, is eliminating single points of failure and disaster recovery is the process of getting a system back to an operational state when a system is rendered inoperative. In essence, disaster recovery picks up when high availability fails, so HA first.”

Vertical Scaling is the process of adding resources to increase power of existing servers. For example, adding more CPUs, adding more RAM, etc.

With vertical scaling alone, the component still remains a single point of failure. In addition, it has hardware limit where if you don’t have more resources, you might not be able to scale vertically.

Databases, cache. It’s common mostly for non-distributed systems.

Horizontal Scaling is the process of adding more resources that will be able handle requests as one unit

A load balancer. You can add more resources, but if you would like them to be part of the process, you have to serve them the requests/responses. Also, data inconsistency is a concern with horizontal scaling.

The load on the producers or consumers may be high which will then cause them to hang or crash.
Instead of working in “push mode”, the consumers can pull tasks only when they are ready to handle them. It can be fixed by using a streaming platform like Kafka, Kinesis, etc. This platform will make sure to handle the high load/traffic and pass tasks/messages to consumers only when the ready to get them.

Take a look here

You can find a list here

Read about it here

Caching is used to speed up read operations by storing frequently accessed data in memory or on a fast storage medium. By keeping data close to the application, caching reduces the latency and overhead of accessing data from a slower, more distant storage system such as a database or disk.

On the other hand, databases are optimized for storing and managing persistent data. Databases are designed to handle concurrent read and write operations, enforce consistency and integrity constraints, and provide features such as indexing and querying.

You can mention:

roll-back & roll-forward cut over dress rehearsals DNS redirection

A central processing unit (CPU) performs basic arithmetic, logic, controlling, and input/output (I/O) operations specified by the instructions in the program. This contrasts with external components such as main memory and I/O circuitry, and specialized processors such as graphics processing units (GPUs).

RAM (Random Access Memory) is the hardware in a computing device where the operating system (OS), application programs and data in current use are kept so they can be quickly reached by the device’s processor. RAM is the main memory in a computer. It is much faster to read from and write to than other kinds of storage, such as a hard disk drive (HDD), solid-state drive (SSD) or optical drive.

An embedded system is a computer system - a combination of a computer processor, computer memory, and input/output peripheral devices—that has a dedicated function within a larger mechanical or electronic system. It is embedded as part of a complete device often including electrical or electronic hardware and mechanical parts.

A common example of an embedded system is a microwave oven’s digital control panel, which is managed by a microcontroller.

When committed to a certain goal, Raspberry Pi can serve as an embedded system.

There are several types of storage, including hard disk drives (HDDs), solid-state drives (SSDs), and optical drives (CD/DVD/Blu-ray). Other types of storage include USB flash drives, memory cards, and network-attached storage (NAS).

Direct memory access (DMA) is a feature of computer systems that allows certain hardware subsystems to access main system memory independently of the central processing unit (CPU).DMA enables devices to share and receive data from the main memory in a computer. It does this while still allowing the CPU to perform other tasks.

A real-time operating system (RTOS) is an operating system (OS) for real-time computing applications that processes data and events that have critically defined time constraints. An RTOS is distinct from a time-sharing operating system, such as Unix, which manages the sharing of system resources with a scheduler, data buffers, or fixed task prioritization in a multitasking or multiprogramming environment. Processing time requirements need to be fully understood and bound rather than just kept as a minimum. All processing must occur within the defined constraints. Real-time operating systems are event-driven and preemptive, meaning the OS can monitor the relevant priority of competing tasks, and make changes to the task priority. Event-driven systems switch between tasks based on their priorities, while time-sharing systems switch the task based on clock interrupts.

There are six classes of interrupts possible:

As defined by Doug Laney:

DataOps seeks to reduce the end-to-end cycle time of data analytics, from the origin of ideas to the literal creation of charts, graphs and models that create value. DataOps combines Agile development, DevOps and statistical process controls and applies them to data analytics.

An answer from talend.com:

“Data architecture is the process of standardizing how organizations collect, store, transform, distribute, and use data. The goal is to deliver relevant data to people who need it, when they need it, and help them make sense of it.”

Wikipedia’s explanation on Data Warehouse Amazon’s explanation on Data Warehouse

Data Lake - Wikipedia

Apache Hadoop - Wikipedia

Responsible for managing the compute resources in clusters and scheduling users’ applications

A programming model for large-scale data processing

HDFS Architecture

1. The client sends a request to the ceph cluster to retrieve data:

   Client could be any of the following

   • Ceph Block Device
   • Ceph Object Gateway
   • Any third party ceph client

2. The client retrieves the latest cluster map from the Ceph Monitor

3. The client uses the CRUSH algorithm to map the object to a placement group. The placement group is then assigned to a OSD.
4. Once the placement group and the OSD Daemon are determined, the client can retrieve the data from the appropriate OSD

1. The client sends a request to the ceph cluster to retrieve data
2. The client retrieves the latest cluster map from the Ceph Monitor
3. The client uses the CRUSH algorithm to map the object to a placement group. The placement group is then assigned to a Ceph OSD Daemon dynamically.
4. The client sends the data to the primary OSD of the determined placement group. If the data is stored in an erasure-coded pool, the primary OSD is responsible for encoding the object into data chunks and coding chunks, and distributing them to the other OSDs.

In general, Packer automates machine images creation. It allows you to focus on configuration prior to deployment while making the images. This allows you start the instances much faster in most cases.

A configuration->deployment which has some advantages like:

This page explains it perfectly:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes
MINOR version when you add functionality in a backwards compatible manner
PATCH version when you make backwards compatible bug fixes
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.