Digital Camera Project

Digital Camera: OV7670 CMOS camera + DE2-115 FPGA board + LandTiger 2.0 board

1. Overview

The objective of this project is to design a simple digital camera system in order to illustrate some of the main concepts related to digital design with VHDL and FPGAs, image and video formats, CMOS cameras, basic image processing algorithms (black and white filters, edge detection, etc.) embedded programming of microcontrollers, and others. This camera is not something that you will be able to compete with against today's sophisticated commercial cameras, but it provides a learning exercise and a platform for you to potentially try out new ideas and enhancements.

2. Main Hardware

We list and discuss here the main hardware components that we use in this project.

2.1. OV7670 CMOS Camera Module

The OV7670 is a CMOS image sensor + DSP that can operate at a maximum of 30 fps and 640 x 480 ("VGA") resolutions, which is the equivalent of 0.3 MegaPixels. The captured image can be pre-processed by the DSP before sending it out. This preprocessing can be configured via the Serial Camera Control Bus (SCCB) or through I2C interface. You can download its datasheet from [2] and implementation guide from [3].

Note that there are actually two versions of this camera module. The simpler and cheaper one is shown in Fig.2.1 - and it is the one we use in this project. However, there is out there also an enhanced version, which, while slightly more expensive (about $20-$30), it has on the back of the module's board a memory chip, which is simply used to buffer image data (see one example in [4]). This enhanced camera module is shown in Fig.2.1; it has some of its access pins different from that in Fig.2.1.


Figure 2.1: OV7070 camera module (used in this project).	Figure 2.2: OV7070 + AL422 FIFO camera module.

The I/O signals of the camera module are listed in the following Fig.2.3. A very nice presentation of how to program this camera module can be found in [5]. Some of that discussion is ported and adapted here.

Signal	Usage	Active
3V3	3.3V power
Gnd	Ground
SIOC	Serial command bus clock (up to 400KHz)
SIOD	Serial command bus data
VSYNC	Vertical Sync	Active High, configurable
HREF	CE output for pixel sampling	Active High, configurable
PCLK	Pixel Clock
XCLK	System clock (10-48MHz, 24MHz Typ)
D0-D7	Pixel data
RESET	Device Reset	Active Low
PWDN	Device Power Down	Active High

Figure 2.3: Description of the I/O signals of the camera module.

A video is a succession of frames. You may want to pause here and read Appendix A (at the bottom of this page), which presents a short discussion of VGA. A frame of the video is a still image taken at an instant of time. A frame is compromised of rows (or lines), and a row is compromised of a number of pixels - whose number on the row equals the number of columns. A pixel is the smallest part of a digital image, and it looks like a colored dot. For example, a simple 4x3 image is shown in Fig.2.4.

	C0	C1	C2	C3
R0	0,0	0,1	0,2	0,3
R1	1,0	1,1	1,3	1,3
R2	2,0	2,1	2,2	2,3

Figure 2.4: Image with 4x3 pixels - four columns x 3 three rows. Each pixel "box" shows the coordinates of the pixel.

There are many pixel formats. Some of the simplest pixel formats include monochrome and RGB. In monochromes images, each pixel is stored as 8 bits, representing gray scale levels from 0 to 255, where 0 is black, 255 is white and the intermediate values are shades of gray. In the RGB color model, any color can be decomposed in Red, Green and Blue light at different intensities. Because a color can be made by mixing Red, Green and Blue, it is called the RGB color system or model. It is also called an "Additive" color system, because it starts at black, and then color is added. Using this model, each pixel must be stored as three intensities of these red, green and blue lights. The most common format is RGB888. In this format each pixel is stored using 24 bits - the red, green and blue channels are stored in 8 bits each:

RRRRRRRRGGGGGGGGBBBBBBBB

For instance, the color red would be stored, in binary, as 24 bits as:

111111110000000000000000

or, as commonly shown in hexadecimal, as:

FF0000

For more information on the RGB format, you can read Wikipedia entry [6].

However, the RGB formats used by the OV7670 are: RGB565, RGB555 and RGB444. The difference, compared to RGB888 format, is the number of bits assigned to each channel. For example, in the RGB565 format, the red channel is stored as 5 bits, the green channel as 6 bits and the blue channel as 5 bits. These formats take less memory when stored but in exchange sacrifice the number of colors available.

Retrieving data from the camera module is relatively simple. As can be seen in the datasheet ([2] pages 7-9), and shown in Fig.2.5 and 2.6, signals VSYNC and HSYNC give us the "points of reference" we need to know when pixel data of a frame are being transmitted from the camera module.


Figure 2.5: VGA frame timing (datasheet of OV7670 camera module).	Figure 2.6: RGB 565 output timing diagram (datasheet of OV7670 camera module).

For example, keeping Fig.2.5 in mind, we would need to do basically the steps described in the following embedded C description in order to get the data for each pixel in a frame.

void read_a_frame_from_ov7670(void)
{
    uint32_t i = 0;

    while (VSYNC); // wait for the old frame to end
    while (!VSYNC); // wait for a new frame to start
    while (VSYNC) {
        while (VSYNC && !HREF); // wait for a row to start
        if (!VSYNC) break; // row did not start, but frame ended
        while (HREF) { // wait for a row to end;
            // first byte
            while (!PCLK); // wait for clock to go high
            // get the first 8 bits, i.e., the first byte
            frame_buffer1[i] = LPC_GPIO2->FIOPIN;
            while (PCLK); // wait for clock to go back low

            // second byte
            while (!PCLK); // wait for clock to go high
            frame_buffer2[i] = LPC_GPIO2->FIOPIN;
            while (PCLK); // wait for clock to go low

            i++;
        }
    }
}

Figure 2.7: Embedded C function to receive two bytes of data (16 bits) for each of the pixels of a frame. This code is written in the context of using the LandTiger 2.0 board for which we write embedded C programs using ARM Keil's IDE. Also, we work with VSYNC negative, which can be set up by programming appropriately first the camera's registers.

2.2. DE2-115 FPGA Development Board

We use the DE2-115 board [7] in this project. This is a really nice board (see Fig.2.8), which has an Altera Cyclone IV E FPGA chip on it, with 114,480 LE (logic elements). It is designed and sold by Terasic [8] and costs about $300. However, with some editing of the VHDL code and constraints files, you should be able to implement this project on other FPGA boards.


Figure 2.8: DE2-115 FPGA board.	Figure 2.9: LandTiger 2.0 board.

To write VHDL code and to synthesize, place & route, generate FPGA programming files, and to actually program the FPGA, we use the freely available Quartus II Web Edition Software from Altera [9]. You should download and install it on your computer; you will need though to register in order to be able to download the installation files.

2.3. LandTiger 2.0 Microcontroller Board

We use this really neat board, LandTiger 2.0 [10], just for its touch screen interface. We send different controls to the main "digital camera entity" (residing on the FPGA) from this board. This board (see Fig.2.9) is built around an NXP LPC1768 microcontroller based on ARM's Cortex-M3 core. It costs around $70, which is an exceptionally good deal for what it has on it. To program this board we use the free edition of the ARM Keil's Microcontroller Development Kit (MDK) IDE, which you can download from [14].

3. Digital Camera - Implementation #1

This is Phase I of this project. Here, we simply build a preliminary design where we simply connect the OV7670 camera module to the FPGA, retrieve video frames from the camera module, store temporarily each image frame's data inside the FPGA, and use that data to drive a VGA monitor connected to the board. We'll complicate this design in follow-up implementations. Note that each implementation is a stand alone design by itself.

This initial implementation is a cleaned up version and ported to the DE2-115 board of the project made publicly available by Mike Field on his website [13]. The block diagram of this first implementation is shown in Fig.3.1.

Figure 3.1: Block diagram of project Implementation #1.

While details of the actual VHDL description can be found in the actual files (entire Quartus II project files can be downloaded at the bottom of this page), the points that are worthwhile mentioning here are:

First, if you do not really know VHDL, fear not, you can start learning it now. I would suggested starting with reading some of the teaching materials available on the course website here [15]. :-)
We use the push button KEY0 of the DE2-115 board. This button is already debounced, so we do not really need to do debouncing inside our VHDL code.
DE2-115 board has an oscillator based clock signal generator which supplies a clock signal of 50 MHz to the FPGA chip. Inside our design, we also need a clock signal of 25 MHz - we get it by using an ALTPLL divider available on the Cyclone IV E. To do that, one must use the Mega-Wizard Plug-In Manager (under Tools menu option) of Quartus II software. You can read more about ALTPLL here [16].
As a frame buffer, we use RAM: 2-PORT mega-function as a Memory Compiler instance that we create again with the Mega-Wizard Plug-In Manager of Quartus II. If you have been using Xilinx ISE before, you could see how to port your VHDL code to Quartus II in this compact application note [17].

4. Digital Camera - Implementation #2

This is Phase II of this project. Here, we complicate a bit things by using one of the two SDRAM chips on the DE2-115 board to save snapshots. The DE2-115 board has two 64Mbyte SDRAM chips (U15 and U13 on the schematic diagram of the board). In this new implementation, we build on top of the implementation #1 to:

Add a second frame buffer
Add an SDRAM controller to connect to one of the two SDRAM chips on the DE2-115 board in order to be able to write to and read from it.
Add two new inputs controlled by the first two push buttons called "take_snapshot" and "display_snapshot". When user presses "take_snapshot", the current frame from buffer 1 should be saved into SDRAM starting at memory address zero. When user presses "display_snapshot", the picture previously saved should be fetched into buffer 2. We display the snapshot from buffer 2 on the VGA monitor while user keeps the "display_snapshot" push button pressed. When none of the take_snapshot or display_snapshot buttons is pressed, the design works in video-mode, where the newly added multiplexer steers frames from buffer 1 to the VGA driver.

Good to remember (when you read the provided VHDL code) is that the DE2-115 board has two SDRAM chips, each chip being 64MB (hence a total of 128MB SDRAM on the board). Each 64MBytes is organized as 8M x 16 x 4 Banks. The 8M (2^3*2^20) is organized as an array of 2^13 = 8192 rows and 2^10 = 1024 columns. Therefore, the address to access any element in such an array of 8192x1024 entries has 23 bits, i.e., address[22:0] (13 bits to index rows and 10 bits to index columns).
The block diagram of this second implementation is shown in Fig.4.1. The entire Quartus II project files can be downloaded at the link provided at the bottom of this page.

Figure 4.1: Simplified block diagram of project Implementation #2.

The SDRAM controller is a port to VHDL from Verilog of the controller implementation from [18].
To understand how the SDRAM controller works as well some of the comments inside the related VHDL code, you should read first the datasheet of the SDRAM chip. You should find it on the Terasic CD that comes with the board. If not, then, you can download it from [19]. Some starting pointers on SDRAM in general can be found at [20].

5. Digital Camera - Implementation #3

This is Phase III of this project. Here we "complicate" things even further in that we add two extra modules to our design. The first module is a grayscale filter [21], which can be applied to the taken snapshot retrieved from SDRAM via buffer 2. The second module is an edge detection entity based on the well known Sobel filter [22]. This filter also works on the image retrieved from the SDRAM memory via buffer 2. The VHDL code of the Sobel filter is adapted from Alessandro's work [23]. The block diagram of this third implementation is shown in Fig.5.1. The entire Quartus II project files can be downloaded at the link provided at the bottom of this page.

Figure 5.1: Simplified block diagram of project Implementation #3.

6. Digital Camera - Implementation #4

This is Phase IV of this project. Here we implement realtime (i.e., dynamic) edge detection video mode. Basically, the video stream from the camera is either displayed normally or first processed as grey-filter + edge-detection and displayed on the VGA monitor. The block diagram of this third implementation is shown in Fig.6.1. The entire Quartus II project files can be downloaded at the link provided at the bottom of this page.

Figure 6.1: Simplified block diagram of project Implementation #4.

APPENDIX A: VGA Basics

Here we discuss briefly the mechanics of Video Graphics Array (VGA). In this project, one way we display video and still images is by driving a VGA monitor. The De2-115 board has a VGA connector, which we use for this purpose.

To drive a VGA monitor, we need to generate a VGA video signal that contains 5 active signals:

Horizontal sync: digital signal, used for synchronization of the video
Vertical sync: digital signal, used for synchronization of the video
Red (R): analog signal (0-0.7 v), used to control the color red
Green (G): analog signal (0-0.7 v), used to control the color green
Blue (B): analog signal (0-0.7 v), used to control the color blue

The display area of the monitor must be scanned in a sequence of horizontal lines, from top to bottom, in order to generate an image (as shown in Fig.A.1). At each pixel location, the RGB color information in the video signal is used to control the color of the pixel. By changing the analog levels of the three RGB signals all other colors are produced.


Figure A.1: Scanning of a frame from pixel 0,0 to pixel 639,479.	Figure A.2: Illustration of Vsync and Hsync signals.

The screen refresh process begins in the top left corner (i.e., pixel 0,0) and "paints" 1 pixel at a time from left to right. At the end of the first row (or line), the row counter increments and the column counter is reset to the first column. One frame is finished when the entire screen has been painted. Then, the refresh process begins again to paint the next frame.

The video signal must redraw the entire screen fast enough to provide for motion in the image and to reduce flicker. The period with which the refreshing is done is called the refresh rate; a value of 60 times per second is good sufficient for the human eye. In the particular case of a VGA display of 640x480 pixels, a 60 Hz refresh rate means about 40 ns per pixel, which can be achieved with a 25 MHz clock signal.

The vertical sync signal tells the monitor to start displaying a new image or frame (as shown in Fig.A.2). The horizontal sync signal tells the monitor to start another row of 640 pixels. In our example of 640x480 VGA mode, after 480 rows of pixels are refreshed with 480 horizontal sync signals, a vertical sync signal resets the monitor to the upper left corner and the process is restarted. In a PC graphics card, a dedicated memory buffer is used to store information about the color value of every pixel in the image or frame. This memory is read out as the "painting beam" scans across the screen to produce the RGB signals.

For more information on VGA, refer to the resource in from references [10,11] or google it.

The DE2-115 board used in this project includes a 15-pin D-SUB connector for VGA output. The VGA synchronization signals must be provided directly from the Cyclone IV E FPGA, and the Analog Devices ADV7123 triple 10-bit high-speed video DAC (only the higher 8-bits are used) is used to produce the analog data signals (red, green, and blue). The ADV7123 chip (see its datasheet here [12]) on the board needs a clock signal, which in our case is the 25 MHz that we will generate inside the FPGA chip by using a clock divider: the 50MHz clock signal provided by the oscillator on board is divided by two. You should identify where this is done inside the provided VHDL code of this project.

REFERENCES

[1] OV7670 CMOS camera module. You can buy it from Amazon and other websites for around $7 (including shipping!). Also, if you google about it, you will find many blog-like sites that describe various interesting projects built around this camera module.
http://www.amazon.com/sunkee-OV7670-640X480-Compatible-Interface/dp/B00AZWVZKW/ref=sr_1_1?s=electronics&ie=UTF8&qid=1379557438&sr=1-1&keywords=CMOS+OV7670+Camera

[2] OV7670 Datasheet (of 2006). This is the simpler and cheaper version, without a FIFO memory on the camera's small board.
http://www.cutedigi.com/pub/sensor/Imaging/OV7670-Datasheet.pdf

[3] OV7670 Implementation Guide and Schematic Diagram.
http://www.haoyuelectronics.com/Attachment/OV7670%20+%20AL422B%28FIFO%29%20Camera%20Module%28V2.0%29/OV7670%20Implementation%20Guide%20%28V1.0%29.pdf
http://www.haoyuelectronics.com/Attachment/OV7670%20+%20AL422B%28FIFO%29%20Camera%20Module%28V2.0%29/OV7670%20+%20AL422B%28FIFO%29%20Camera%20Module%28V2.0%29%20Schematic.pdf

[4] OV7670 AL422 FIFO Camera Module.
http://www.ebay.com/itm/OV7670-AL422-FIFO-Camera-Module-/270755438712?_trksid=p2054897.l4275

[5] Jorge Aparicio's blog entry on the OV7670 camera module. Very nice presentation of programming and of basic image formats too. Uses a STM32F4 microcontroller to get the image data from the camera module and then send it via serial port to the host PC, where image is displayed within a simple Python based interface.
http://embeddedprogrammer.blogspot.com/2012/07/hacking-ov7670-camera-module-sccb-cheat.html

[6] Wikipedia entry on RGB color model.
http://en.wikipedia.org/wiki/RGB_color_model

[7] DE2-115 FPGA development board.
http://www.altera.com/education/univ/materials/boards/de2-115/unv-de2-115-board.html

[8] Terasic website.
http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&No=502

[9] Quartus II Web Edition Software
http://www.altera.com/products/software/quartus-ii/web-edition/qts-we-index.html

[10] Video Graphics Array, VGA (Wikipedia entry).
http://en.wikipedia.org/wiki/Video_Graphics_Array

[11] Eduardo Sanchez, A VGA display controller.
http://lslwww.epfl.ch/pages/teaching/cours_lsl/ca_es/VGA.pdf

[12] ADV7123 Datasheet.
http://www.analog.com/static/imported-files/data_sheets/ADV7123.pdf

[13] Mike Field's VHDL design using OV7670 camera module. The image data from camera is buffered inside the FPGA (Xilinx's chips on Nexys2 or Zedboard) and then displayed on an VGA monitor attached to the board. This VHDL implementation represented the initial/starting point in this project.
http://hamsterworks.co.nz/mediawiki/index.php/OV7670_camera

[14] ARM Keil's MDK IDE.
http://www.keil.com/arm/mdk.asp

[15] Cristinel Ababei's HDL Based Digital Design with Programmable Logic course materials.
http://dejazzer.com/ee478/index.html

[16] Phase-Locked Loop Reconfiguration (ALTPLL_RECONFIG) IP Core. User Guide.
http://www.altera.com/literature/ug/ug_altpll.pdf

[17] AN 307: Altera Design Flow for Xilinx Users.
http://www.altera.com/literature/an/an307.pdf

[18] Original Verilog implementation of the SDRAM controller.
http://whoyouvotefor.info/altera_sdram.shtml

[19] Datasheet of SDRAM chips available on DE2-115 board. Part number IS45S16320B.
http://www.issi.com/WW/pdf/42S16320B-86400B.pdf
[20] Synchronous dynamic random-access memory (SDRAM) entry on Wikipedia.
http://en.wikipedia.org/wiki/Synchronous_dynamic_random-access_memory

[21] Grayscale pointers:
Grayscale (Wikipedia's brief presentation of grayscale)
Conversion from color to grayscale ("averaging" method)

[22] Edge detection - Sobel Operator (Wikipedia's introduction)
[23] Alessandro Montanari, Sobel Operator in VHDL, 2011. https://code.google.com/p/vhdl-project

[24] Compression (JPEG) - Digital Camera Example (Ch.7 in Frank Vahid's textbook; look up for the slides)

Complete Implementations Downloads
Quartus II project files of this project, implementation #1
Quartus II project files of this project, implementation #2
Quartus II project files of this project, implementation #3
Quartus II project files of this project, implementation #4