Crio Bytes - Fun with Process Internals | Crio.Do | Project-Based Learning Platform for Developers

Objective

Learn to appreciate the capabilities of the proc filesystem by fetching process internals and doing some cool things.

Background

The /proc filesystem (or Procfs) is an encyclopedia of system and process related information. This acts as a tunnel to the kernel space data without the overhead of system calls to access it. Usual commands like ps and sysctl utilise Procfs to obtain information and change kernel parameters at runtime respectively.

The numbered folders represent entries for a process denoted by its process id and the named folders contain system-related information

image alt text

/proc/self contains data on the current process

Some of the important folders to look out for, given a process id, are:

fd - file descriptors of the process - contains one entry for each file opened by the process

image alt text

fdinfo - contains info on open file descriptors like file offset, permission etc

image alt text

maps - memory mapping of the process - lists the memory mapping of different segments of the process like heap, stack etc

image alt text

mem - virtual memory of the process - we can see that it has write permissions. Ooh!

image alt text

Primary goals

Understand different ways in which procfs can be utilized to get process internals

Objective

Learn to appreciate the capabilities of the proc filesystem by fetching process internals and doing some cool things.

Background

The numbered folders represent entries for a process denoted by its process id and the named folders contain system-related information

image alt text

/proc/self contains data on the current process

Some of the important folders to look out for, given a process id, are:

fd - file descriptors of the process - contains one entry for each file opened by the process

image alt text

fdinfo - contains info on open file descriptors like file offset, permission etc

image alt text

maps - memory mapping of the process - lists the memory mapping of different segments of the process like heap, stack etc

image alt text

mem - virtual memory of the process - we can see that it has write permissions. Ooh!

image alt text

Primary goals

Understand different ways in which procfs can be utilized to get process internals

Getting Started

You need access to a Linux machine with sudo access.
Have g++ compiler to run simple cpp programs.


 g++ SampleProgram.cc -o SampleProgram

./SampleProgram

Understand how to run a background process and get its process id.

image alt text

When you run a program with ‘&’ in the end, it runs as a background job and prints the process id. In the above case, 226285 is the process id.

You may have to periodically kill these processes you put in the background. Otherwise your system may become slow. If you run ps in the same terminal, you will be able to see the list of all processes. You can then kill the process either using pkill or kill commands.

image alt text

What to do if an important file is deleted?

Linux doesn’t delete files that are still open by a process. We'll be restoring an accidentally deleted file utilising this newly acquired wisdom

Create a dummy file with some text (some text => lorem ipsum)


echo "Lorem ipsum dolor sit amet, consectetur adipiscing elit" > lorem_ipsum.txt

Let's start a program that reads the contents of the lorem_ipsum.txt file very slowly


//File: SlowReader.cc

//Compilation: g++ SlowReader.cc -o SlowReader


#include<iostream>

#include<fstream>

#include<unistd.h>


using namespace std;


char buffer[1];


int main(int argc, char** argv) {

	if (argc != 2) {

		cout << "Usage: ./SlowReader <fileToRead>" << endl;

		return 0;

	}


	fstream fin;

	fin.open(argv[1]);


	cout << "Reading " << argv[1] << " slowly." << flush;


	while(fin.read(buffer, sizeof(buffer))) {

		cout << "." << flush;

		sleep(30);

	}


	fin.close();

	cout << endl << "Read complete" << endl;


}

image alt text

Now, trick yourselves into deleting lorem_ipsum.txt and verify your negligence with the ls command

A couple of things we know

Linux didn't actually delete the file
We have enough time until our SlowReader process terminates

So, let's check the procfs file descriptor info for our process

image alt text

Aha! There's one pointing to our deceased file. Let's copy that and check it’s contents to verify

image alt text

Now you know how to prank your friend by deleting his/her project presentation when it's being used :)

(NB: No credits taken for any harm caused by this knowledge)

Curious Cats

What are those extra file descriptors? We had only opened a single file, right?
What would happen if the process runs to completion while the copy command was still executing? Will the file be partially copied or no file is transferred?
As we were running a dummy program in the background, we had enough time. What can we do quickly if the process was running in the foreground & would complete before we get to find the PID, file descriptor and copy the file?
Does the Proc file-system exist on disk like other files?

Examining file read operation

Programming languages allows us to read file contents in chunks. This is really important when dealing with large files that may not fit into the system memory. We can provide how many bytes to read at a time. Internally, does the OS or the programming language really read in the number of bytes we provide as parameters?

Start by creating a larger file, 100KB would do

image alt text

Run the SlowReader2 program and feed it large.txt. The program is instructed to read 2048 bytes at a time into the buffer.


//File: SlowReader2.cc

//Compilation: g++ SlowReader2.cc -o SlowReader2


#include<iostream>

#include<fstream>

#include<unistd.h>


using namespace std;


char buffer[2048];


int main(int argc, char** argv) {

	if (argc != 2) {

		cout << "Usage: ./SlowReader2 <fileToRead>" << endl;

		return 0;

	}


	fstream fin;

	fin.open(argv[1]);


	cout << "Reading " << argv[1] << " slowly." << flush;


	while(fin.read(buffer, sizeof(buffer))) {

		cout << "." << flush;

		sleep(1);

	}


	fin.close();

	cout << endl << "Read complete" << endl;


}

image alt text

Get the file descriptor for our large.txt file

image alt text

Now if we check the contents of /proc/[pid]/fdinfo/[fd], the pos attribute denotes the file read offset for the process

image alt text

Ok, so the program has read 24573 bytes (out of 102400) for processing.

Try printing out the values repeatedly to spot how it changes. Does it increment by the buffer size we’ve set in the SlowReader program? Who manipulated the buffer read size we set? OS or C++ or Both?

See here

Curious Cats

If we have the same file opened twice in a process at the same time, will there be a common file descriptor or two? Why is it so?
Use the pos values in /proc/[pid]/fdinfo to output progress of processing the file. How about a progress bar itself?

Hint: dialog --gauge command can be used to show a progress bar in bash

Can we allocate more than physical memory?

How much memory can we allocate for a process? Can we allocate more than the system physical memory? Let’s see.

This program goes on allocating memory up to 10GB.


// File: MemEater.cc


#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>


 int main () {  

         int n = 0;  


         while (n < 10000) {  

                 if (malloc(1<<20) == NULL) {  

                         printf("malloc failure after %d MiB\n", n);  

                         return 0;  

                 }   

	     n++;

          }  

        printf ("Allocated %d MiB\n", n);

        sleep(300);

 }

The program runs without any issues. You don’t have 10 GB RAM, right?

image alt text

Now, if we check the process’s memory usage with the top command, it’s only using around 40MB of RAM and we can see 9.8GB allocated to virtual memory instead

image alt text

So, what happens is that the Linux machine goes on overpromising memory to processes, beyond what it has, without actually reserving space in the RAM.

Now, let’s try to write some contents to the allocated memory and see if there’s any difference. We’ll allocate and use 300MB of memory.


// File: MemUser.cc

// Only to be run inside a virtual machine

#include <stdio.h>

#include <string.h>

#include <stdlib.h>

#include<unistd.h>

 int main (void) {  

         int n = 0;  

         char *p;  


         while (n < 300) {  

                 if ((p = (char*)malloc(1<<20)) == NULL) {  

                         printf("malloc failure after %d MiB\n", n);  

                         return 0;  

                 }  

                 memset (p, 0, (1<<20));

	     n++; 

         }	

         printf ("Used %d MiB\n",n);

         sleep(300);

 }

Now, if you check the memory usage by the process, it will be somewhere around 300000KB (300MB). So, the process is actually using memory now.

image alt text

Two parameters in the Procfs determine whether to overcommit memory and how much

image alt text

overcommit_memory flag determines if to overly commit memory to processes than available if possible (MemEater program above) & overcommit_ratio is up to how much percentage of physical memory to allocate

These fields are writable. If overcommit_memory is set to 2, OS doesn’t overcommit memory.

image alt text

Source

What is a disadvantage of overcommitting memory?

Curious Cats

Try playing around with different values for overcommit_memory & overcommit_ratio. See if you can trick the OS to allocate even more memory.
Here we only had our MemEater/MemUser process as the memory-intensive processes running. What would happen in a more usual setting where we have multiple applications like our browser, video player etc. running together? How does the OS choose the process to be killed? (Hint: OOM Score - /proc/[pid]/oom)
Start a number of applications that you know will take up considerable memory space and check their OOM Score. How does it compare to less memory thirsty applications?

References

Hacking the virtual memory

We saw while prepping up with Procfs earlier in the overview that the virtual memory mapping (/proc/[pid]/mem) was writable. How about we try to change the process parameters?

Run this program and enter a text which we’ll be modifying by wiring into its virtual memory


// File: Program3.cc

#include<unistd.h>

#include<stdio.h>

#include<bits/stdc++.h>

using namespace std;


int main(int argc, char** argv) {

	int textSize = 0;

	char *ptr = new char[30];

	string msg;


	strcpy(ptr, argv[1]);

	int i = 0;

	while(1) {

		printf("%d: %s @ %p \n", ++i, (char*)ptr, (void *)ptr);

		sleep(3);	

	}


}

image alt text

The text we entered is stored starting from this virtual memory address 0x5629b8046e70

We have the virtual memory allocation table courtesy of /proc/[pid]/maps. Let’s peek into that to have a better sense of where exactly it resides

image alt text

Okay, the initial addresses contain the data and code segments of the process, virtual addresses from 5629b8035000-5629b8056000 are part of the heap and within 7ffe9671d000-7ffe9673e000, we have the stack area. So, our text at 0x5629b8046e70 resides in the heap which is expected, as we have dynamically allocated memory using the new keyword.

We can use a program to overwrite the content of memory space in the process’ virtual address space. This takes in the PID, address of the variable and the text to replace with. Run this program in a new terminal using sudo. Why sudo?


// File: overwrite_vm.cc


#include<stdio.h>

#include<fstream>

#include<string>

#include<sstream>

using namespace std;


int main(int argc, char** argv) {


	stringstream s;

	s << "/proc/" << argv[1] << "/mem";

	string mem_filepath = s.str();


	fstream f(mem_filepath.c_str(), ios::in | ios::out | ios::binary);

	perror("Open status");

	

	int addr_start;

	s.str("");

	s << argv[2];

	s >> hex >> addr_start;


	string replace_msg(argv[3]);

	string end_char = "\0";

	string msg = replace_msg + end_char;


	f.seekp(addr_start, ios::beg);

	f.write(msg.c_str(), sizeof(msg));

	perror("Replace status");


	f.close();	


	return 0;

}

image alt text

Voila!

image alt text

One advantage with this method was that we didn’t need to stop the program to change a variable value used by it. So, can this be used to update parameters for processes we can’t afford to stop and then restart?

Tip

On 32-bit machines, addresses are only 32-bits

A hex digit - 0xf = 1111 -> 4 bits

So 8 digits are there in the address - 0x26a7c010

For 64-bit systems, addresses are 64-bits

0x7fdc81399010 -> Check if they have 16 digits. If not, why?

Curious Cats

Update the code to take in a search string instead of the address itself, find the address by searching the string in the heap space and then replace the string
We have the data and code segments in the first part of the virtual memory. Can the program itself be re-written while running?

Newfound Superpowers

Knowledge of an all powerful Proc Filesystem that controls the Unix/Linux world.
Each directory or file under Procfs has a significant meaning. You have explored some and will continue to explore others.

Now you can

Monitor from behind the scenes what a process is doing
Answer questions about Procfs in a way you couldn’t before
Write your own System Monitor (cpu and memory used, files open etc.) by using contents of Procfs
Explain better why random file access is slower than its sequential counterpart
Understand how you are able to run applications like your favorite video game that’s larger than the size of the RAM

Curious Cats

Write a bash script to reproduce the output of the ps command by fetching data from the procfs (Hint: /proc/[pid]/stat)
We saw how to access and modify memory intentionally. But, how does the OS check against normal processes stepping into each other's memory?
Network related files as well are present in the Procfs. Can you utilize this to block ping commands to your system? (Hint: /proc/sys/net/ipv4)
Figure out how malicious programs can reside in our system without showing themselves in the file system

Byte Introduction

Objective

Background

Primary goals

Objective

Background

Primary goals

Getting Started

What to do if an important file is deleted?

Curious Cats

Examining file read operation

Curious Cats

Can we allocate more than physical memory?

Curious Cats

References

Hacking the virtual memory

Tip

Curious Cats

Newfound Superpowers

Now you can

Curious Cats