>...collaboration with the industry to develop a camera stack... protecting vendor core IP.
Can't say I have any interest in protecting "vendor IP" in their crappy camera drivers. Let vendors open their blobs! Closed source camera drivers will just reduce quality and increase bugginess.
The point is to get high quality software that actually belongs to us all. Accepting closed source drivers means accepting the Windows model where companies release buggy drivers for one or two combinations of processor architecture and operating system version before ending all support. The company gets to open wash their reputation while still keeping us dependent on them for scraps of support. Giving us nothing at all would've been more respectful than this.
Letting companies protect their intellectual property benefits absolutely nobody but them. They don't have our best interests in mind, only their own. The ideal solution is to reverse engineer their hardware's interface and produce a competing implementation which will no doubt be better than their minimum viable product. My laptop has a backlit keyboard with RGB LEDs and I managed to figure out how it works and make my own program to replace the manufacturer's. I'm so happy I don't have to boot into Windows just to use their slow software anymore.
They don't even have to give us source code. Documentation would eliminate the need to actually reverse engineer the hardware. They could just release that data and all interested parties would start hacking. The problem would take care of itself at no cost to them.
> What's the point of this all or nothing attitude?
the point is to try to enact change in corporate behavior with regards to consumer devices.
>The alternative is getting nothing.
goes both ways.
If I don't buy a product that uses binary blobs all over the place, they don't sell the product to me.
Supporting products that support DRM systems natively does a lot to support the underlying DRM system, too. It sets a native expectation of DRM among developers, fostering the use.
And worst of all, I can't try to fix blobs without labor that is so intensive that it renders the very idea moot.
Because all or nothing forces someone to compromise. By default almost all vendors would pick to keep things proprietary. If going open source is the only possible option then some vendors will release the source and if they don't then hobbyists will write their own drivers since the work is very useful.
Wikipedia faced a similar issue when considering using https everywhere. Some countries were blocking certain pages on Wikipedia and https would break that so it was uncertain if they would just block the whole of Wikipedia now. As it turns out they mostly did not block the whole website and those selected pages became unblocked because the alternative was given by the all or nothing approach was too far.
> Because all or nothing forces someone to compromise
When exactly did that work? In which project? All this attitude did was to make Linux unusable for many people, making it marginal and thus a smaller power at pushing companies to support and opensource drivers.
Things started changing AFTER more people were able to adopt it due to less black and white attitude of some distributions.
There are technical reasons for this but it also serves another purpose: maintaining leverage over companies that would take advantage. The Linux kernel receives tens of patches every hour. If companies refuse to upstream their drivers, they are left behind and must pay the maintenance costs or drop support. Their products are worse for it.
I wish every free software project had this power.
A goog example is Open WRT - the project basicaly got started due to LinkSys finally fulfilling their GPL obligation and releasing drivers to one of their Linux based wireless routers.
Because when we were more "all or nothing", corps were meeting us halfway and starting to open their drivers. Pragmatism works today, but sacrifices tomorrow.
The biggest issue is that "raw" Linux is not an RTOS.
When the photographer presses the shutter, you have a few milliseconds to capture the image. You can't have the OS decide to prioritize receiving a tweet before getting to the shutter release.
It's even more intense with video, and most still cameras are quickly morphing into high-quality video cameras.
Also, image processing is REAL intense; especially with Computational Photography. It can suck down your battery in seconds.
There's a few ways to deal with it. You could use something like Xenomai or RTLinux, or establish some kind of SoC for the realtime-sensitive processing (what most phone manufacturers probably do -Apple sure does).
I'm not sure I understand what you mean. I've done my share of video and photo programming on Linux (in particular I've written non-trivial v4l2 drivers for use in things like car rear view cameras, where obviously low latency is absolutely critical and you have to resort to tricks like processing the image by slice before you've read the full sensor) but I'm not sure I see how that's relevant for a "point and shoot" type camera.
For one thing I don't think any reasonable camera would have the CPU "bitbang" the reading of the sensor's pixel, it will be read and buffered in hardware and set to the system RAM through DMA. We're talking about hundreds of megabytes of raw data per picture here, it's not something you want to memcpy(), especially on an embedded processor. If receiving a tweet is enough to make you miss some pixels then you haven't prioritized your RAM controller correctly or your system is severely overloaded already.
The shutter release is independent of course, but it's not something where a few microseconds are going to change anything, especially for a mechanical shutter where you need to move physical parts. If your CPU stalls for 1ms processing a tweet then, again, you have bigger problems. You don't need an RTOS to control a camera shutter.
So that leaves us with the problem of latency but even there I fail to see the problem. My girlfriend is a professional photographer and when she shoots she can either get the preview on the camera's screen (but that involves switching to the preview which already usually takes a pair of seconds) or getting the image on a PC connected through USB, which also takes a significant amount of time. Adding or removing 100ms of latency won't change a lot in practice as far as I can tell.
The "receiving a tweet" thing was really just a strawman. It's more likely to be issues of OS maintenance that could cause the problems.
I've had some really relevant experience, here, myself. I won't go into detail (N of the D-A).
> it will be read and buffered in hardware and set to the system RAM through DMA.
Moving memory, even on the fastest processor, is always a cost. This is especially true, when you look at the pixel count of most cameras (not just DSLRs), these days. Also, you don't just get the memory at once in a big multi-megabyte dump. The sensor needs to be polled. Each cell is queried for its charge state in a linear fashion. There may be ways to access sections in parallel, but I haven't encountered it (It would make sense -I have tended to be on the software side of things, leaving the hardware to the electrical engineering wizards). This is usually dealt with by an ASIC connected directly to the sensor.
If you want to have a "software-only" camera (as opposed to one with a dedicated SoC), then you are looking at some really heavy-duty challenges.
The idea of a "software-only" camera, is to completely do away with the dedicated SoC, or replace it with one that can be re-flashed.
Not so simple.
Computational Photography is something that is necessary for consumer-grade cameras, and is increasingly becoming relevant to pros, as well. It takes some real horsepower.
The newer iPhones have awesome cameras. I suspect that some of the higher-end Androids are just as good, or better. If so, I will bet that you'll find a custom ASIC in the woodpile.
Apple has custom ASICs for image handling, as well as CP. They have an ML chip built into their phones.
I am confident that these challenges will be overcome (there's money to be made, and that's always a great driver), but the challenges are real. There are some very, very smart folks that have beat their brains out on this.
The other part of this, is that the entire image processing/video arena is a swamp of patents.
They are quite enforceable ones, too. Video has been around quite a while, and used to be the domain of big fish corporations that could easily afford to file patents.
You need to have something like <50ms frame lag, generally. I know there's some studies out there about this. It's been a while since I've done this stuff, but there's an "uncanny valley" of lag that needs to be avoided.
Also, you need to present an image that is useful to the photographer.
For example, if you are applying vignette correction, that can be a processor-intensive step, and can be difficult to apply at 24 frames per second on a full HD image (the sensor may be returning considerably higher than that, but the camera back may just have a 1080p display).
If you are doing things like image enhancement/CP (for example smart HDR, selective enhancement, or stochastic NR), then how will they be displayed? Do you show a grainy image in the preview, even though the final will look much better? What about color space conversion? The display is likely to have a limited gamut, but the image may be saved at full Adobe, and you want to have perceptual correction, as opposed to clipped.
This is an exciting time to be looking at this stuff, but the challenges are real.
In order to capture an image, most image sensors are triggered. Some sensors will operate in "free-run" mode, but in reality something is sending a capture signal to the camera. In sensors where you have very low level control over things, usually what that does is clear the pixels (dump the accumulated charge) and start exposing. Then, when the exposure is complete, the camera makes the data available for read-out (for a phone sensor analogue-digital conversion is likely done in circuitry on the sensor, not in the phone).
That trigger is the important bit. Using triggering you can synchronise multiple cameras (e.g. the sensors in multi-camera phones are almost certainly co-triggered to avoid ghosting), illumination units, etc. In fact many camera chips can also be set to output flash signals when they're exposing, this eliminates latency in sending separate "capture" and "flash" signals. When I've deployed machine vision systems, often we would use an external system to trigger everything (e.g. a microcontroller). Interfacing hardware and software triggering is a nightmare.
Then you've got whatever low level interface is actually communicating to the camera. On a phone, that's probably MIPI/CSI-2. Modern SoCs have dedicated (hardware) camera interfaces (the ISP) that are designed specifically to load images from camera sensors and do some minimal processing. You need kernel-level drivers for whatever your SoC is. These drivers will (in theory) let you set up arbitrary image sensors with whatever resolution/timing they require. But the key is you need two drivers: one to control the ISP and another to send commands to the camera sensor. A lot of the time (cough Broadcom) you only get a blob for the ISP and you almost never get full datasheets for image sensors without being a big customer.
Want to make an educated guess why libcamera supports the [Rockchip] RK3399? They make it comparatively easy.
In the end V4L acts as a mediator between this low level interface and other things which want to talk to the camera. Rockchip have provided what amounts to a plugin for V4L. Then on top of V4L you have libraries like OpenCV which give you nice interfaces like `imread` to capture images and do stuff with them. To answer your question you probably need to be more specific. The performance of what V4L does for a specific ISP/sensor combination very much depends on those underlying drivers.
If you can afford it, a safe strategy is to leave the sensor in free-running mode in a separate thread that's dedicated to capturing images. Then in your application you pick off the most recent frame and do stuff with it in parallel. For a 30fps camera you therefore have 30ms or so to do your stuff before the next frame comes in.
Not sure. Never played with it. My experience is with a custom Android build, on some standard Snapdragon stuff (won't get any more detailed than that).
I have played with ffmpeg, which is not suitable for mobile, unless heavily modified. The VideoLAN folks seem to have gotten VLC working for mobile, but it's a real pig.
If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs.
Given how hard other manufacturers try and fail to make equally good cameras, it's going to be a steep climb for the open source community too.
So far I am not seeing much work being done in that space. Maybe this is a start?
I want an open non-spying phone but it's going to be hard to let go of the amazing camera in my pocket now (Pixel 3a).
Not sure why you'd need a great camera in a phone. Normal sensor will be just fine for most use cases. I have a phone without a camera, and I don't miss it either.
It's much easier and better to use a camcoder to capture video. Easier to hold in your hand, easier to handle, less shaking, physical buttons, wrist strap, more control, larger sensor, faster to turn on and start recording, etc. I'd never exchange it for a phone for this purpose.
> "If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs."
Couldn't they just find another niche? I think relatively few consumers know anything about photography. Even a proper professional DSLR would be useless to me because I know next to nothing about photography and tools don't make the artist. No picture I've ever taken has any artistic merit or technical merit.
Of course, good camera software is a good thing and for the segment of society that's enthused about photography it's going to be essential. I'm just not convinced it's the secret sauce to mass appeal.
Edit for response:
> Well, okay, what problem would you rather see them tackle?
I don't have such a preference. I expect volunteer developers will work on matters they find personally compelling. Those that find photography software compelling should pursue their passion.
The reason for a good camera is not to take artistic photography (although you can do that too and many do).
It's because people use that camera to record memories that they cherish. Just ask any new parent what happens to their phone's camera roll once number 1 pops out. Or ask young people to show you where they went on their last vacation: many times it'll be a bunch of cool shots taken on a phone.
These memories are priceless and having to record them with a device that is open but has the photo quality of a 5 year old device is not an attractive trade-off.
> I think relatively few consumers know anything about photography.
That's precisely why you need good image processing software that works automatically. Nearly everybody wants to take great photos with their phone, but few people know how to configure everything themselves, and even those who do probably don't want to fiddle with every shot.
> Couldn't they just find another niche?
Well, okay, what problem would you rather see them tackle?
> Of course, good camera software is a good thing and for the segment of society that's enthused about photography it's going to be essential. I'm just not convinced it's the secret sauce to mass appeal.
We already have the hard data: almost without exception, people who use phones want their phones to take nice pictures.
For reasoning about camera software, it is entirely reasonable to assume that “the segment of society that is enthused about photography” is basically the exact same segment as “people who have phones”.
If you don’t believe me, show me a popular smartphone that doesn’t have a camera. :)
I can show you a lot of people who rarely if ever use their smartphones to take pictures, and more still who primarily take pictures for utilitarian purposes (such as preserving the information on a whiteboard before erasing it, remembering where they parked their car, etc.) So I definitely don't agree that "owns a phone" and "enthused about photography" are basically the same groups of people. The former is much larger than the later.
> If you don’t believe me, show me a popular smartphone that doesn’t have a camera. :)
I can show you plenty that lack good cameras... I don't think that's a compelling argument anyway. iphones have barometers but that doesn't mean everybody who owns one cares about their altitude or predicting the weather.
> such as preserving the information on a whiteboard before erasing it, remembering where they parked their car, etc.) So I definitely don't agree that "owns a phone
You’re an engineer that doesn’t go on a lot of vacations then. My mom can see the difference between the photo quality of her iPhone 6 and my dad’s iPhone 11, and she’s everything but an expert.
If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs.
This is a view without nuance.
Lots of people don't use, or only rarely use, a phone camera. It also implies that [GNU|other-free]/Linux "winning" a substantial share of the market is a plausible (or even desirable) outcome.
[GNU|other-free]/Linux is a terrible phone operating system for the majority of people. I say this as a person who has a PinePhone that's about to ship (and is planning on using it as my daily driver). I also take hundreds of pictures a year. The trick to solve the camera problem is just buying a good camera that isn't your phone.
If [GNU|other-free]/Linux gets 10% of the phone operating system market, the amount of malware (both in the literal sense and the Stallmanic sense) written for it will skyrocket. That's certainly not desirable, and at that point it's more or less an Android clone, which defeats the point.
The "average" user doesn't understand that they shouldn't download software from any website that offers it, for example. Android is already bad at sandboxing and keeping less-technical users safe. Imagine if less-technical users got their hands on a mobile [GNU|other-free]/Linux distribution. Chaos.
By foregoing a high-quality camera, it limits the market share of a device like this. That's good! Non-technical users should be kept very far away from it until it's baby-proofed or education improves. Anything else will harm them.
Freeing them is a good goal, but it probably won't come first, and probably won't come for a while. It's especially not something that should have the roadmap shifted around to match.
(The separation between "Linux" and "OS/Linux" is relevant here, because without it, there's no distinction between it and Android.)
> Lots of people don't use, or only rarely use, a phone camera.
Huh...? This couldn't be further from the truth. They may not be terribly picky about the technical details about them, but actual human beings take photos... all the damned time.
> The trick to solve the camera problem is just buying a good camera that isn't your phone.
Coupled with your other view on this, it's not worth discussing how ridiculously inconvenient that would be for what people take photos for: social media.
This may not reflect your own usage pattern, but it sure as hell is what real people do with their phones. Taking pictures of their casseroles and making duck lips with their friends.
Twitter only has something like 70,000,000 active U.S. users, Facebook only has 244,000,000 in North America and Canada combined. That's nowhere near the entire U.S. population, and probably half of those take photos with their phones.
Any operating system that plans to stop supporting the hardware on devices worth hundreds of dollars (sometimes more) after a mere two to three years is deplorable in my book. I really dislike the iOS experience and its lack of customization, and I find the disposable upgrade cycle of Android phones incredibly wasteful.
Meanwhile, I'm happily running Fedora on a 2011 MBP with the latest security updates. I can't wait until Linux phones reach the level of support and security of laptops and desktops.
I'm surprised to see such big emphasis on support for Android. Most camera modules come with their own drivers that already provide Android support. One benefit could be the licensing, but after a quick inspection it is unclear to me what license this library is under -- there is a licenses folder with 4 different licenses in addition to a developer agreement.
The design seems to be heavily inspired by the Android camera API: per-frame configuration, 3A, multiple stream support, device enumeration, etc.
> The HAL will implement internally features required by Android and missing from libcamera, such as JPEG encoding support.
That is interesting, since most camera modules will have a hardware accelerated path to encode frames directly to JPEG. If it's done internally, it will be much slower than all other implementations I'm aware of.
> the Linux media community has very recently started collaboration with the industry to develop a camera stack that will be open-source-friendly while still protecting vendor core IP
What does that mean? Open source should be open source.
Is it going to work on top of v4l or it's independent?
I don't get who is going to be using this library.
Most android developers would prefer to use native Java APIs instead of a C++ library ( with a JNI wrapper around it?) since it's easier. Linux phones barely have any market share so this library would be used mainly on computers where most of the cameras are USB anyway.
Interesting. Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera? Would it support receiving a picture from eg a flatbed scanner over a parallel port?
> Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera?
I don't know details about the Raspberry Pi cameras, but if it's a UVC camera (quite likely) then it should be supported by this library according to the documentation. A USB Logitech camera should also be supported. The gotcha is that unless those cameras expose advanced functionality (3A, multiple stream support, etc) then you don't get any benefit over using the standard V4L drivers.
> Would it support receiving a picture from eg a flatbed scanner over a parallel port?
I don't think scanners identify themselves as camera devices, but someone can correct me if I'm wrong.
I'd expect it to be complementary to the existing libraries for scanning/cameras, not mutually exclusive. So something like sane or libgphoto would presumably link in libcamera to access those "complicated" devices.
Can't say I have any interest in protecting "vendor IP" in their crappy camera drivers. Let vendors open their blobs! Closed source camera drivers will just reduce quality and increase bugginess.