Automatic Parallel interview

When I started my PhD. -thesis a couple of years ago, I took some time to look at auto-parallelizing compilers and research. after all, I wanted to work on making parallel programming easier, and the best way to do that wowould surely be to let compilers do all
The work. unfortunately, the field appeared to be quite dead at that time. there has been a huge amount of research done in the eighties and nineties, yet it all appeared to have settled down. and the compilers I tried cocould not parallelize more than the simplest
Loops. I have always been asking myself why this was the case, and when I had the chance to talk to dr. jay hoeflinger, he had some very interesting answers for me. he agreed to let me re-ask these questions in an email interview and this is the result. thanks,
Jay, for sharing your knowledge!

Michael:First of all, please share a little bit about your biography and background on automatic parallation with us!

Jay:I have a BS (1974), MS (1977) and PhD (1998) from the University of Illinois at Urbana-Champaign. I filled in the large gap between the MS and the PhD with varous things. I did some scientific programming, some business
Programming, and then in 1985 I joined the center for supercomputing Research and Development (csrd) at the University of Illinois. I worked on the cedar Fortran parallelizing compiler, which was being built to drive
The Cedar supercomputer being built at csrd. we had high hopes that a parallelizing compiler wocould be the "silver bullet" that wocould allow an easy way to program parallel machines, which we viewed as being inevitable.

We made some big strides in understanding the challenges facing parallelizing compilers, but by the time csrd lost its funding, we were far from solving the problems.

After csrd, I stayed at the University of Illinois and worked on the Polaris parallelizing compiler. extends talented students contributed to Polaris, and it had good success over the years. I contributed to the compiler and was involved in your papers about
The work. finally I decided to work on a PhD degree, attacking the problem of helping the compiler represent the complex data access patterns of programs. I thought that the only way to really solve the automation parallation problem was to give the Compiler
A richer data structure for representing Data Accesses in a program.

I finally got the PhD degree in 1998, but it was obvious at that time that automatic parallation by the compiler alone was not near a solution. A richer data structure for the compiler helped, but still the compiler had too little information about
Program. the only hope was to do something at runtime. of course, giving the compiler A richer way to represent complex Data Accesses forms a basis for that, and that work is finding some success, but a general solution of the problem remains out of reach.

In 1998 I went to work for the Center for simulation of advanced rockets as a Senior Research Scientist. I studied various aspects of parallering ing the rocket simulation codes being developed at the center.

In 2000, I joined Intel and have worked on the OpenMP Runtime Library and the cluster OpenMP Runtime Library. while working for intel I also have been involved in the OpenMP 2.0, 2.5 and 3.0 language committees. i'm a senior staff software engineer and
Team lead of
Cluster OpenMP project.

Michael:Wow, now I know why you areThe man(TM) to talk to about automatic IC parallelization. Let us start
Actual interview with the current state of automatically parallelizing compilers: What are they able to do today?

Jay:Vectorization technology and simple parallelization has moved into mainstream extends cial compilers. This is because the students that worked on parallation ation in the 1980 s and 90 s now form the core of various extends cial
Compiler teams. lost of the techniques used to attack parallallocation ation have moved into specified cial compilers and are supporting both parallel and serial optimization.

Today, parallpolicing compilers can be fairly successful with FORTRAN77 programs. the language is simple, making it more possible to represent what the program is doing in the compiler's internal data structures. success with C has been more limited. in
90 s a lot of work was done on parallelizing Java programs, with some success. In the last 10 years work has been done on parallelizing domain-specific languages, like Matlab.

As people realized how hard the parallelization of programs at compile time was, they started turning to parallelizing at runtime. under projects have studied this problem and there has been some success. parallelization at runtime has the distinct advantage
That all the information about variable values is available. some of the complicated mechanisms that were developed to symbolically simulate execution in the compiler and propagate data and program attributes throughout the program aren't needed at runtime.

Now, research compilers can work together with the runtime system to parallelize a code. for instance, if a compiler is trying to prove a certain relation in order to parallelize a given program loop, and is unable to prove it, the compiler cocould compile
A runtime test into the program that can determine whether the relation is true. parallequalation decisions can then be based on the result of the test.

Michael:OK, now that we know what they can do let's talk about the more important part: What are they not able do? And what are the reasons for that?

Jay:The key to parallallocation ation is finding a way to divide up the work of the Code such that a write to a memory location by one thread is always separated from another thread's access to the memory location by a synchronization.
So, uncertainty about which data is being accessed by which threads really hampers parallelization.

This uncertainty can result from data values not being available to the compiler, for instance, if they are read from an input file at runtime or simply calculated at runtime based on a complex formula. this means that compilers have difficulty with Adaptive
Codes that adjust the way they access data, based on complicated functions or external information. some very clever techniques have been devised to allow the compiler to parallelize even in these complex situations, but typically such techniques require long
Compile times.

In the specified cial world, compile time is much more important than it is in the research world. so, complicated techniques that can parallelize loops in a research compiler by causing compilation to take a long time, may not be implemented in a commercial
Compiler-at least they wocould not be used by default. So, commercial compilers are usually limited to rather simple parallation techniques.

Achieving the same success in C or C ++ as we see in FORTRAN77 is much more difficult because of the presence of pointers. pointers create a big problem for compilers because they obscure what data is actually being accessed, making the problem of understanding
Data access even harder than it wocould be otherwise.

Michael:Is there hope? Compiler writers are some of the smartest people in computer science and parallel programming is a huge problem to attack, what kind of help can we hope for from our compilers in the near future?

Jay:I think there is some hope in limited domains. there is some hope for parallelizing domain-specific ages, for instance. MATLAB wocould be an example of a language that is great for linear algebra, but not so much
Other things. A Matlab compiler doesn't have to consider all the things that a C ++ compiler has to consider.

The thing you have to understand is that compilers are enormous programs, typically written by a large group of experts. even without automatic IC parallation, the modern specified cial compiler is extremely complex, and has a very large amount of source code.
The software engineering challenges of design and testing for such a large program are already daunting. adding the large amount of source code that wocould be required to do a really good job of parallallocation ation wocould only exacerbate this problem.

So, I think it is unlikely that compilers in the near future will do a significantly better job of parallelization. this is especially true due to the rising popularity of Explicitly Parallel ages and the increasing willingness of universities to teach
People about parallel algorithms and how to design their codes for parallelism from the start.

I believe that compiler vendors are more interested in increasing support for the Explicitly Parallel ages than they are in trying to do automatic parallation.

Michael:Cocould a change in ages help? You mentioned earlier that Fortran 77 code is easier to optimize/parallelize for compilers than e.g. c/C ++ code. cocould a switch to a more restricted/more regular or just different language (maybe
Even a new one) change the situation and allow our compilers to help us more?

Jay:There is a tension between the expressibility of a language and the ability of compilers to properly optimize it. A language with a high degree of expressibility may be easier for humans to use, yet at the same time it
Almost certainly is more complex and therefore more difficult for a compiler to understand, making it harder to parallelize.

A program written in a language with limited expressibility (e.g. FORTRAN77) is therefore easier to parallelize, yet harder for a person to write programs. defining a new language that's sufficiently rich in expressibility, yet simple enough to be routinely
Successfully parallinclued is a difficult challenge. It seems possible, but no good solution yet.

This is why OpenMP has become popular. openMP has the serial program embedded in it, but allows the programmer to tell the compiler where the parallel parts are. the programmer's knowledge of the Code is put to its best use. the programmer typically knows
That conceptually, a given region of the program consists of independent computation, and can just say so in a very simple way to the compiler.

Michael:Where do you see automatically parallelizing compilers in 5 years from now? Will some of the problems you have already sketched be solved by then? Any silver bullets on the horizon? Please take a look into your crystal ball
!

Jay:I think there's hope for domain-specific compilers as I explained above. There may be some hope for runtime parallation ation using virtual machines or maybe just runtime tests compiled into the code.

But, for the most part, parallel code will still need to be designed and written by experts. along these lines, I think parallel libraries will be much more prominent in 5 years. these libraries allow people to write serial programs that call the library
Routines to solve various problems in parallel, in which the Library Routines themselves have been optimized by experts.

So, the emphasis has to shift to people writing their own parallel code, not relying on compilers. Universities will need to do a better job of producing parallel experts.

For the most part, I think automatic IC parallconfiguration will remain an unfulfilled dream.

So, unfortunately, I don't see any silver bullets. Automatic IC parallallocation ation remains an extremely interesting research problem, though. Maybe one day, when someone much smarter than I has a great idea...

Michael:Thank you, Jay, that was very enlightening! And although the outlook at the end may sound pessimistic, I am looking forward to interviewing the smart guy you have mentioned. You never know-he may be right around the corner implementing
His revolutionary idea in a compiler as we speak...