------------------------------------------------------------------------------- Reading a file into memory Simple way -- auto allocating Use getdelim() (related to getline()) to read in a file up to the first NUL or EOF is reached. For text files there is no 'NUL' so this should read in the whole file, plus a final teminating NUL character. FILE *file = stdin; char *data = NULL; size_t size = 0; size_t nchars; nchars = getdelim(&data, &size, 0, file); if (ferror(file)) { /* handle error */ } ... free(data); See similar Example (using getline()) http://pubs.opengroup.org/onlinepubs/9699919799/functions/getdelim.html Implementation source... https://github.com/bwhite/hadoopy/blob/master/hadoopy/getdelim.c https://gist.github.com/ingramj/1105106 --------- Direct use of read() to read in whole files. NOTES: the file print is not good with binary files! Has extra unneeded memory copies, and reallocation is not doubling read() does not add a teminating NULL char. #include #include #include #include int main () { char buf[4096]; ssize_t n; char *str = NULL; size_t len = 0; while (n = read(STDIN_FILENO, buf, sizeof buf)) { if (n < 0) { if (errno == EAGAIN) continue; perror("read"); break; } str = realloc(str, len + n + 1); memcpy(str + len, buf, n); len += n; str[len] = '\0'; } printf("%.*s\n", len, str); free(str); return 0; } --------- Reallocate larger buffers as needed.. From Stack exchange Code review http://codereview.stackexchange.com/questions/82268/ NOTE: fread() does not add a terminating NUL char itself. #define BUFFER_SIZE 4096 /* Read whole file into memory */ int read_file(const char *filename) { assert(fd); FILE *file; char *file_data = NULL; size_t file_size = 0; size_t buffer_size = getpagesize(); /* memory page size, typically 4kb */ size_t read_chars; if(filename) { file = fopen(filename, "r"); } else { file = stdin; } assert( file == NULL ); /* open failure */ while(1) { /* allocate */ file_data = (char *)realloc(file_data, buffer_size); assert(file_data); /* allocate error */ /* read */ read_chars = fread(&file_data[file_size], 1, buffer_size-file_size, file); assert (ferror(file) == 0); /* read error */ file_size += read_chars; /* loop exit */ if ( feof(file) ) break; /* double the buffer realloc() buffer_size -- if we loop */ buffer_size *= 2; /* optional - add a maximum buffer_size limit error */ } assert(file_size); /* empty file error */ /* do what you need with file_data, and file_size */ free(file_data); } ------------------------------------------------------------------------------- For a good example on using select() see "select.example" ------------------------------------------------------------------------------- Dis-associate from the Controlling TTY if( (tty=open("/dev/tty", O_RDWR)) >= 0 ) { (void)ioctl( tty, TIOCNOTTY, 0 ); (void)close( tty ); } ------------------------------------------------------------------------------- Socket Processing (with select) What seems to be happening is this: I'm reading and writing sockets merrily, and detect when clients go away by reading zero bytes after select() told me the socket was readable. This handles the situation when the process on the other end of the socket exits or is aborted. However, the client can go away more abruptly: in our case, say, somebody slices the phone line or unplugs the ethernet connector. In this situation, I don't get the normal indication of a client going away. Rather, my end of the socket does nothing, then 10 minutes later, my program exits (evidently TCP is doing the exit). No core file is left. It just ends, without me telling it to do so. Solution: Try setting setsockopt(fd,SOL_SOCKET, SO_KEEPALIVE), this will cause you to get a select with hangup or error within several seconds of the line loss. Your exit issue sounds like a bug. Does your OS have a trace capability like SYSV.4 (truss) to trace the system calls after your line loss? cgi@crl.com (Paul Smith) If connectivity is lost you shouldn't find out about it unless (1) you're exchanging data with the other end, in which case the 10 minute eventual timeout is typical for TCP, or (2) you've set the keepalive option (which takes a default of 2 hours to detect the loss). rstevens@noao.edu (W. Richard Stevens) ------------------------------------------------------------------------------- Interupts with Write() and XDR communication I would simply write data blocks with the "write()" system call, and if a signal interrupted the write (on a System V machine, BSD machines restart the write automatically), I would simply loop back and use the return value to see how much was written and try to write out the rest. This worked wonderfully. Using XDR to make the communication more portable between machines. The problem is, XDR routines simply return TRUE upon success and FALSE upon failure. And they fail if interrupted by a signal. As far as I'm concerned, the implementation of the xdr primitives should check the return value of write(), just as I had done, and continue the operation transparently to the person using the primitive, so that the primitive returns TRUE even if a signal occurred somewhere along the way. Unfortunately, this does not happen. (As least not on an SGI running Irix 4.x) Solution: What I did was to use XDR into a block in memory using xdrmem_* routines. This block of memory occurred just after a (4 byte) unsigned long which was set to contain the length of the block plus itself. The length was manually htonl'd. (Or maybe I used xdr_ulong?) When writing, I always wrote the block out including the prefixed length, and kept attempting to write what was left before returning, so that nothing else got written in the middle. When reading, I always read the length separately (retrying as necessary to get the length). After I had my 4 bytes, I ntohl'd (or xdr_ulong'd?) it, and then kept attempting to read the remainder of the block. Once I had it, I could then XDR the stuff out. If I got EOF on my socket before reading all, I knew to close it (and stop write attempts, too). ------------------------------------------------------------------------------- POpen for both read and write The problem with trying to pipe both input and output to an arbitrary slave process is that deadlock can occur, if both processes are waiting for not-yet-generated input at the same time. Deadlock can be avoided only by having BOTH sides follow a strict deadlock-free protocol, but since that requires cooperation from the processes it is inappropriate for a popen()-like library function. The 'expect' distribution includes a library of functions that a C programmer can call directly. One of the functions does the equivalent of a popen for both reading and writing. It uses ptys rather than pipes, and has no deadlock problem. ------------------------------------------------------------------------------- Example program of doing a pipe. /* pipe.c: exec two commands in a pipe */ #define NULL (char *)0 #define MAXA 16 main(argc, argv) int argc; char *argv[]; { char *av1[MAXA], *av2[MAXA]; int i, n, p[2], cpid; i = 0; n = 0; while ( ++i < argc && n < MAXA ) { if ( *argv[i] == '|' && *(argv[i]+1) == '\0' ) break; av1[n++] = argv[i]; } if ( n == 0 ) uexit(); av1[n] = NULL; n = 0; while ( ++i < argc && n < MAXA ) av2[n++] = argv[i]; if ( n == 0 ) uexit(); av2[n] = NULL; if ( pipe(p) != 0 ) exit(1); if ( ( cpid = fork() ) == (-1) ) exit(1); else if ( cpid == 0 ) { (void)close(p[0]); (void)close(1); (void)dup(p[1]); (void)close(p[1]); (void)execv(av1[0], av1); _exit(127); } else { (void)close(p[1]); (void)close(0); (void)dup(p[0]); (void)close(p[0]); (void)execv(av2[0], av2); _exit(127); } /*NOTREACHED*/ } uexit() { (void)write(2, "Usage: pipe | \n", 34); exit(1); } ------------------------------------------------------------------------------- Writing... You should check for errors after write or fflush; if errno isn't either EINTR or EAGAIN you should generally stop writing to the pipe for good. Note that EPIPE is one of those errors, which you get if a SIGPIPE siggnal doesn't terminate the process. You can also use send instead of write to write to a socket. If you use the MSG_NOSIGNAL flag for send, the SIGPIPE signal will not be sent: the call will just return (ssize_t)-1 with errno set to EPIPE. ------------------------------------------------------------------------------- tail -f processing open file forever read data if none available then sleep 1 else display data fi -------------------------------------------------------------------------------