Friday, September 30, 2011
Capturing the Webcam in Silverlight
Introduction
Since the first release of Silverlight, the community has been patiently waiting for a version of Silverlight which would allow us to capture video from a user’s webcam, as well as audio from their microphone. This past month at PDC 09, our wait was over with the release of Silverlight 4 Beta 1.In this article we will explore the ways in which Silverlight can interact with a webcam, as well as the powerful results one can produce by combining live video with pixel shaders, video brushes as well as WriteableBitmaps.
Download Source Code
Getting Started
To get started, let’s create a new Silverlight 4 project in Visual Studio 2010 and draw a 320x240 pixel black Rectangle called “rectVideo” inside of the LayoutRoot of our MainPage.xaml.
One of the coolest media features included in Silverlight, is the VideoBrush. A VideoBrush gives us the ability to paint any area of our stage with video content. Back in the earlier versions of Silverlight we were limited to painting the source of a MediaElement, such as a Video Stream being played from an external source. A lot of people used the VideoBrush as a way to create cool effects like reflections of a video playing.
With Silverlight 4 addition of Webcam support, we now have the ability to set the source of our VideoBrush to that of the live video being captured through our webcam. We will use this technique to display our video on the Rectangle we just added to our stage.
Attaching to our Webcam
In order to have some live video to display in our black rectangle, we need to attach to our camera (VideoCaptureDevice) and capture its source. The first step in doing this is to instantiate the System.Windows.Media.CaptureSource object. This class is designed to provide methods and properties used to work with audio and video capture devices. In essence, CaptureSource is like a little media player. We attach a device, such as the camera or microphone, and “Start()” or “Stop()” the capture of content through the device. CaptureSource does not give us direct access to the raw audio or video, but instead can be used as the source of a VideoBrush or a custom VideoSink. In the future we will explore how to leverage a class derived from VideoSink to receive video information and to obtain the capture graph.The first step in setting up our CaptureSource, is to set its “VideoCaptureDevice” and “AudioCaptureDevice” properties equal to one of our attached audio or video devices. Silverlight has created a helper class called “CaptureDeviceConfiguration” which gives us two choices in how we can get access to a camera or microphone.
All available Camera’s or Microphones
The first option we have is to call CaptureDeviceConfiguration.GetAvailableVideoCaptureDevices. This method will return us a generic Collection containing each available VideoCaptureDevice. From this collection we could allow our users to manually select which device they choose to capture or we could look for the device which has the “IsDefaultDevice” property set to true.Default Camera or Microphone
A second approach we can use is to request the default VideoCaptureDevice . Most people only have a single camera attached to their computer, so requesting the default will get us access without an additional interaction required by the user. Below is the code I use to attach our default camera to my CaptureSource and use that source to feed live video into my VideoBrush. CaptureSource _capture.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();VideoBrush videoBrush = new VideoBrush();
videoBrush.Stretch = Stretch.Uniform; videoBrush.SetSource(_capture); rectVideo.Fill = videoBrush;Requesting Access
Now that we have our webcam attached, and our rectangle’s Fill being painted by our VideoBrush, it time to request permission to use the camera. For privacy reasons each time that Silverlight wants to access ones camera or microphone, the user must explicitly give Silverlight permission.
To get this dialog to appear we need to make the following call. The first part of the condition looks to see if we have already been given access, while the second requests the dialog to appear. A true response from either will allow us to call our CaptureSource.Start() method to begin capturing our live video.
if (CaptureDeviceConfiguration.AllowedDeviceAccess ||
CaptureDeviceConfiguration.RequestDeviceAccess()) { _capture.Start(); }
Taking it to the next level
Now that we know how to display our webcam in Silverlight, let’s see what we can do with the video being captured.In Silverlight 3 we saw the introduction of Pixel Shader’s as a way to provide a custom bitmap effect to an image or video. Creating your own ShaderEffect is a great way to dynamically transform our webcams video as it is being painted on our rectangle. Instead of spending a bunch of time talking about how to create a ShaderEffect I encourage everyone to go to http://wpffx.codeplex.com/ and download the Windows Presentation Foundation Pixel Shader Effects Library. It contains just about every effect one would want to apply to our video.
To apply a ShaderEffect to our rectangle, we can either do it procedurally or simply add an Effect node to our XAML.
<Rectangle Name="rectVideo" Width="320" Height="240" Fill="Black"
MouseLeftButtonDown="rectVideo_MouseLeftButton" VerticalAlignment="Top">
<Rectangle.Effect> <local:InvertColorEffect/> </Rectangle.Effect> </Rectangle>In this example I have applied an Invert effect to my video producing the following results. As each frame of the video is painted to the rectangle the ShaderEffect is applied just prior to the render.
If I want to give my user access to a bunch of effects, then on cool approach is to create a custom list that previews the effect on a small thumbnail of the live video. Using the same VideoBrush I can paint my webcam video on small 48x36 rectangles that are being rendered with one of each of my ShaderEffects. If a user wants to see the effect on the larger video, then we can tie a MouseLeftButtonDown event on the rectangle that alters our larger rectVideo to display the effect.
private void Effect_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
{Rectangle rec = sender as Rectangle;
switch (rec.Name)
{case "invert":
var invert = new InvertColorEffect();
rectVideo.Effect = invert;break;
case "mono":
var mono = new MonochromeEffect();
mono.FilterColor = Colors.White; rectVideo.Effect = mono;break;
case "swirl":
var swirl = new SwirlEffect();
swirl.SwirlStrength = 2.5; rectVideo.Effect = swirl;break;
case "tone":
var tone = new ColorToneEffect();
tone.Toned = 2; tone.LightColor = Colors.Brown; rectVideo.Effect = tone;break;
case "emboss":
var emboss = new EmbossedEffect();
emboss.Amount = 15; emboss.Width = .01; rectVideo.Effect = emboss;break;
default:
rectVideo.Effect = null;
break;
} }
Capturing Stills of our Live Video
Now that we have our Shader’s being applied to our video, It would be really fun to capture still images that we can store both in isolated storage and to our file system.In most situations, the fastest and easiest way to capture a still is to use the AsyncCaptureImage method exposed on our CaptureSource.
_capture.AsyncCaptureImage((capturedImage) => _viewModel.Captures.Add(capturedImage));Unfortunately, since this technique goes directly against the CaptureSource, it does not reflect the ShaderEffect that we have applied to our rectangle which has been painted with our VideoBrush. To get our still to have the effect, we simply have to generate the WritableBitmap from rectVideo directly. One possible downside is that the size of the still we capture will be the exact size of the rectangle we have drawn. If this was a problem we could certainly explore other ways to capture a large image using a similar approach.
WriteableBitmap writeableBitmap = new WriteableBitmap(rectVideo, null);
string name = Guid.NewGuid().ToString() + ".jpg";
//store the image in a collection in my viewmodel
_viewModel.Captures.Add(new Capture() { Name = name, Bitmap = writeableBitmap });
Isolated Storage
One of the other cool features I decided to add to my example, is the ability to create a list of thumbnails of each still I capture. This collection of thumbnails is displayed in a WrapPanel that has been applied to a ListBox as its ItemPanelTemplate. As I capture the still, I add it to my ListBox via binding to an “ObservableCollection”, as well as writing the image as a jpeg in isolated storage. This allows the application to maintain a persistent cache of each photo that has been captured. Next time we load the Silverlight app the list of stored image will be redisplayed in the ListBox.
Saving Still as Jpeg to File System
Out of the box Silverlight does not have native support of encoding jpegs, but a nice open source library called FJCore has been create that does the trick. http://code.google.com/p/fjcore/using (IsolatedStorageFileStream isfs = new
IsolatedStorageFileStream(name, FileMode.CreateNew, _isf)) {MemoryStream stream = new MemoryStream();
writeableBitmap.EncodeJpeg(stream); stream.CopyTo(isfs); }private void Button_Click(object sender, RoutedEventArgs e)
{Capture capture = listImages.SelectedItem as Capture;
if (capture != null)
{if (_saveFileDlg.ShowDialog().Value)
{using (Stream stream = _saveFileDlg.OpenFile())
{if (_isf.FileExists(capture.Name))
{using (IsolatedStorageFileStream isfs =
new IsolatedStorageFileStream(capture.Name, FileMode.Open, _isf))
