Valve und die zukünftigen Pläne mit VR

SolKutTeR · 12. Mai 2018

Sorry, aber ich verstehe beim besten willen nicht den Sinn der Aufregung, im Zusammenspiel des Themas, des Videos und den enthaltenen leaks.

SolKutTeR · 15. Januar 2019

Kleines Recap an Neuerungen, welche wir im letzten Jahr auf der Steam-Ebene, z.B. im Bezug auf VR erhielten.
Leider keine Auflistung an Tools, welche im letztem Jahr die Steam Plattform bereichert haben, wie z.B. ein fpsVR.

Steam: Das Jahr 2018 im Rückblick
Am Ende eines jeden Jahres nehmen wir uns gerne Zeit, um die Arbeit der letzten zwölf Monate zu bewerten und Pläne für die Zukunft zu schmieden. Dies ist ein informeller, interner Prozess und etwas, das wir normalerweise nicht veröffentlichen. Aber bei einem näheren Blick auf die Arbeit, die wir im Jahr 2018 in Steam investiert haben, stellten wir fest, dass unsere Partner nicht alles davon sehen können. Daher dachten wir, dass es interessant sein könnte, dieses Jahr einige dieser Informationen mit Ihnen zu teilen.

Weitere Themen:

Der Shop

Die Steam Community

Weitere Spielmöglichkeiten

Weitere Steamworks-Funktionen

Hinter den Kulissen

2019 und darüber hinaus

Weiterlesen →

Hier der VR-Abschnitt:

2018 haben wir High-End-VR-Erfahrungen für PC auf vielfältige Weise verbessert und weiterentwickelt. Wir haben SteamVR-Eingabe eingeführt, damit Entwickler über eine Eingabe-Abstraktionsebene verfügen und damit Sie sich nicht um all die verschiedenen Controller-Typen kümmern müssen, die SteamVR unterstützt (zusätzlich haben den SteamVR Unity Plugin aktualisiert, um SteamVR-Eingabe zu unterstützen). Im Anschluss daran haben wir SteamVR Skeletteingabe eingeführt. Mit dieser Funktion können VR-Controller die Handposition des Nutzers so genau erfassen, wie es ihre Sensoren erlauben und diese dann gebündelt als Animationsdaten an VR-Anwendungen weiterleiten. Mit diesen Informationen können Entwickler wiederum die Hände des Avatars und deren Interaktionen in der VR-Welt animieren. Im November haben wir SteamVR mit Bewegungsglättung herausgebracht, wodurch mehr Spieler auf mehr PCs hochwertige VR-Spiele erleben können. Wenn SteamVR feststellt, dass eine Anwendung die Bildfrequenz nicht erreicht, also Einzelbilder auslässt, kommt die Bewegungsglättung zum Einsatz. Die Funktion vergleicht die zwei letzten Bilder, schätzt Bewegungen und Animationen ab, und extrapoliert ein neues Zwischenbild.

SteamVR-Startort verfügt nach mehreren Aktualisierungen jetzt über die Funktion neue Steamfelder zu erzeugen und ermöglicht es den Erstellern von Inhalten Asset-Pakete für die Gestaltung ihrer Umgebungen zu verwenden (hurra!). Außerdem haben wir weitere Inhalte für SteamVR-Startort veröffentlicht, darunter Sammelgegenstände für Spiele wie Moss und Skyrim VR und Umgebungen wie das Team Fortress 2 Supervillain Lair, das The Candy Emporium und Winter Peak.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

SolKutTeR · 24. März 2019

Bei der Valve Keynote auf der GDC bereits angesprochen,.. das Braininterface.

Hier ein Artikel von RoadtoVR

Valve Psychologist: Brain-computer Interfaces Are Coming & Could Be Built into VR Headsets
Valve’s resident experimental psychologist Dr. Mike Ambinder took the stage at GDC 2019 today for his talk on the state of brain-computer interfaces (BCIs) and how they’ll inform the future of game design. In his talk, Ambinder put forth a near future where VR/AR headsets are kitted with non-invasive electroencephalogram (EEG) devices that could one day provide data to game designers so they can create a new generation of smarter, more reactive games. The long-term view is definitely trending towards neuronal implants, Ambinder says, but we’re not there yet. Weiterlesen →

SolKutTeR · 30. März 2019

Interessante Aufbereitung von dem Reddit User olemartinorg im Bezug auf weitere Patente.
(Ob wir das eine oder andere in der Index finden werden?)

Valve patents: speculation about the Index HMD

Hi all!

We're all stoked to see the upcoming Index HMD. A few days ago I was digging around in Valve patents (that have been published after the Vive came out), and I think there are some nice tidbits in those that may indicate some of the features coming in the Valve Index.

In chronological order of published date (all are links to the patent PDFs):
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

1 - Mar. 23, 2017 - PLAYER BIOFEEDBACK FOR DYNAMICALLY CONTROLLING A VIDEO GAME STATE

This has been talked about a lot before, and isn't really VR specific. VNN has a video on brain interfacing recently. A VR HMD is already something you put on your head, so it's not that out-there to think that some future HMD may have sensors that can figure out your current emotional state and adjust the game based on that.

This is starting off slightly off-topic here, but I just want to put out some links here. Mary Lou Jepsen worked at Facebook/Oculus for about a year back in 2015-2016 and has worked on lots of interesting display technology before. She went straight from Facebook to start up OpenWater that develops technology that can enable MRI-like brain imaging from just infrared light, but with far superior resolution, and in a form factor that would fit a headband. See her TED talk about it here. GabeN has said (and VNN quotes) that "[computer-brain-interfaces] are a lot further along than most people anticipate". Maybe this is it? I wonder if that's an addon that comes later, and is what the expansion port thing on the front is for. Also, look at this patent (not by Valve, though).
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

2 - Nov. 28, 2017 - DISPLAY WITH STACKED EMISSION AND CONTROL LOGIC LAYERS

Some interesting part from the patent on what this means in practice (emphasis mine):

Corresponding advantages of silicon electron mobility and processing for display fabrication allows a clear and immediate pathway to retinal near eye displays (e.g., dimensions of 10,000x10,000 or more), high dynamic range, greatly improved fill factor, and brightness (e.g., 10,000 cd/m2 or more). In addition, with respect to production of a display panel, significant improvements in pixel density for significantly larger sizes of display panel may be achieved at significantly lower costs (e.g., $50 or less). In addition, the described techniques may provide a variety of benefits with respect to the use of such a display panel, including to provide one or more of the following: increased bandwidth in communication of video signals to the display panel, photorealistic immersive visual experience [...].
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

3 - Feb. 8, 2018 - MITIGATION OF SCREEN DOOR EFFECT IN HEAD - MOUNTED DISPLAYS

This is pretty straight-forward. This one is talking about using a microlens display in between the lens and the users eye in order to reduce the screen door effect. As far as I understand it, it's like having an array of tiny magnifying glasses so that you'll see more of the pixels and less of the space in between the pixels. It differs from the diffusion filter used in the Odyssey+, as that was placed between the panels and the lens, not between the lens and your eyes (as this patent is about). It probably also wouldn't reduce the sharpness of the display, as I understand it.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

4 - Feb. 22, 2018 - SYSTEMS AND METHODS FOR DETECTION AND / OR CORRECTION OF PIXEL LUMINOSITY AND / OR CHROMINANCE RESPONSE VARIATION IN DISPLAYS

This is something to use in the production process in order to reduce visual artifacts that vary between individual displays and from pixel to pixel. By having the GPU (or an on-board chip) modify the video signal to reverse the artifacts for the given display, it works kind-of like noise cancellation, but for brightness and color "noise" in the display, thus reducing mura.

Interesting side note about putting this workload in the GPU or on an on-board chip (again, emphasis mine):

In certain HMD - related embodiments, mura correction processing in accordance with aspects of the present invention is performed host - side on the graphics processing unit ("GPU"). However, depending on the requirements of each particular implementation, such processing may be effected in silicon, in the headset itself, on a tether, or in the display panel electronics, for example. Such alternative implementations may provide greater image compressibility, which is important in situations involving limited link bandwidths (e.g., wireless systems).

Although it shouldn't come as a surprise that Valve has at least toyed with the thought of a wireless system.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

5 - Mar. 22, 2018 - OPTICAL SYSTEM FOR HEAD - MOUNTED DISPLAY SYSTEM

This one was a bit hard for me to understand. It talks about using double layer fresnel lenses in an HMD in a way that provides multiple "fields" (e.g., small FOV angles, large FOV angles), and allows for seamless transitions when looking between these zones. Figure 3 is of particular interest, I think, as it indicates how this can be used to create an HMD with a massively larger FOV than one could with only one fresnel layer.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

6 - May 3, 2018 - USING PUPIL LOCATION TO CORRECT OPTICAL LENS DISTORTION

This thing describes itself. Using eye tracking to correct for optical distortions. The XTAL HMD already has automatic IPD adjustment based on their eye tracking, but this is going beyond that by configuring optimal lens distortion mapping based on where your pupils are in three dimensional space in front of the lenses.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

All in all, very much to be exited about. All of this leads me to believe that Valve has made the most graphically stunning VR HMD to date. It might not have eye tracking or wireless (yet?), but I do think it will at least have substantially reduced screen door effect and higher FOV. What do you think?
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

SolKutTeR · 18. März 2020

Interessantes Interview von IGN mit Gabe Newell himself, über Half-Life und die weiteren Schritte/Bestrebungen von Valve.

In a rare half-hour conversation, Valve co-founder Gabe Newell talks about the studio's past with Half-Life, the surprising answer to what his favorite Half-Life game is, mistakes that Valve has made, what keeps a man worth billions of dollars coming to work every day, how he spends his days, what the next decade looks like for Valve (hint: it involves a brain-AI interface), and more!
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

SolKutTeR · 19. Juni 2020

Große Ausschreibung bei Valve hinsichtlich Software Engineer für VR & Hardware, interessante Stellen welche dort ausgeschrieben sind:

Software Engineer for VR & Hardware
At Valve we are pushing the boundaries of virtual reality and related technologies. We are looking for versatile, self-directed software engineers in many disciplines that can help us achieve the next steps in VR. Join the Valve team pursuing hardware design, software development, prototyping, testing, and production as we invent, design, and deliver new experiences to over 100 million customers.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

Qualifications - One or More of the Following

User Interfaces

Human Computer Interaction

Controllers

Computer Vision (Tracking, Photogrammetry, Bundle Adjustment)

Image-Based Rendering & Light Field Rendering

Optics & Computational Displays

Tracking (EKF, State Estimation, Inertial Navigation, Control Theory)

3D Graphics: Shading, Rendering, Drivers

Engine Integration (Unity, Unreal)

Spatialized Audio

Human Perception & Vision Science

FPGA / ASIC Development

Analog / Mixed Signal Circuit Design

Embedded Firmware

Video Compression

Wireless Technologies

Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

Auf auf und bewerben, um die VR Zukunft weiter voranzubringen.

SolKutTeR · 18. Juli 2020

Neues Eye-Tracking Patent seitens Valve, welches auf eine dynamische Renderpipeline aufbaut.
(Somit der erste Schritt dahingehend SteamVR die neue Technik beizubringen.)

Dynamic Render Time Targeting Based On Eye Tracking
A head-mounted display (HMD) with a rolling illumination display panel can dynamically target a render time for a given frame based on eye tracking. Using this approach, re-projection adjustments are minimized at the location of the display(s) where the user is looking, which mitigates unwanted, re-projection-based visual artifacts in that “region of interest.” For example, logic of the HMD may predict a location on the display panel where a user will be looking during an illumination time period for a given frame, determine a time, within that illumination time period, at which an individual subset of the pixels that corresponds to the predicted location will be illuminated, predict a pose that the HMD will be in at the determined time, and send pose data indicative of this predicted pose to an application for rendering the frame.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

CROSS REFERENCE TO RELATED APPLICATION
[0001] This application claims priority to commonly assigned, co-pending U.S. Provisional Patent Application Ser. No. 62/792,305, filed Jan. 14, 2019. Application Ser. No. 62/792,305 is fully incorporated herein by reference.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

BACKGROUND
[0002] Virtual reality (VR) systems are used both within and outside of the video game industry. Displays for VR systems, such as those embedded in a VR headset, can operate using various display driving techniques. Some VR systems use a “global flashing” technique to drive the display(s). In the global flashing technique, all of the pixels of the display(s) illuminate in synchronization to present an image on the display(s) for a given frame. Other VR systems use a “rolling band” technique to drive the display(s). In the rolling band technique, rows of pixels are illuminated in sequence (typically from top-to-bottom) to present an image on the display(s) for a given frame. In other words, with the rolling band display driving technique, the illumination propagates (or “rolls”) across the display during the presentation of an image for a given frame.

[0003] A user of a VR headset that implements the rolling band display driving technique may, when looking at the top or the bottom of the display(s), notice unwanted visual artifacts, such as “judder,” with respect to moving or animating objects in those regions. Judder causes the user to perceive a “double ghosting effect” where a moving object–such as a virtual representation of the user’s hand moving at the bottom of the display–appears to bounce between two locations (or separate from itself) frame-to-frame. The cause of this unwanted visual artifact near the top and bottom of the display(s) is partly due to the target render time that is provided to the application for rendering a given frame, and partly due to a re-projection step that applies rotational corrections to the application-rendered frame to present an image that is aligned with the most up-to-date estimate of the user’s head pose.

[0004] Consider a graphics-based application, such as a video game, which outputs rendered frames that are used to present images on the display panel(s) of a VR headset. In order to render the correct scene based on the user’s head pose, the application receives, as input, a predicted pose of the user’s head. However, with a rasterization-based rendering approach, the application can only accept a single pose (based on a single target render time) as input for rendering the frame in question. This means that a single target render time must be chosen for the original pose prediction of the user’s head. One option, in a VR system that uses the rolling band display driving technique, is to choose, as the target render time for every frame, the time when the middle row of pixels will be illuminated. Although this approach is expected to minimize the total amount of re-projection adjustments for the entire frame, it nevertheless fails to mitigate the unwanted judder artifacts that are noticed by the user near the top and the bottom of the display(s) because the user is not always looking at the middle of the display panel(s). In fact, the user’s eyes tend to wander about the display panel(s), often gazing at the top or the bottom of the display(s) as the VR headset is used. When the user’s eyes stray from the middle of the display to the top or the bottom of the display, for example, the re-projection adjustments that are applied to the pixel values near the top or the bottom of the display(s) can cause noticeable judder artifacts with respect to moving or animating objects in those regions of the display.

[0005] Provided herein are technical solutions to improve and enhance these and other systems.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

BRIEF DESCRIPTION OF THE DRAWINGS
[0006] The detailed description is described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0007] FIG. 1 is a diagram illustrating an example technique for dynamically targeting a render time for a given frame based on eye tracking, in accordance with embodiments disclosed herein.

[0008] FIG. 2 is a diagram illustrating how the degree of re-projection adjustments varies across the display for different subsets of pixels when using an approach to dynamically target the render time for a given frame based on eye tracking, in accordance with embodiments disclosed herein.

[0009] FIG. 3 illustrates a flow diagram of an example process for dynamically targeting a render time for a given frame based on eye tracking, in accordance with embodiments disclosed herein.

[0010] FIG. 4 illustrates a flow diagram of an example sub-process for applying re-projection adjustments, in accordance with embodiments disclosed herein.

[0011] FIG. 5 illustrates example components of a wearable device, such as a VR headset, in which the techniques disclosed herein can be implemented.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

DETAILED DESCRIPTION
[0012] Described herein are, among other things, techniques and systems, including a head-mounted display (HMD) system, for dynamically targeting a render time for a given frame based on eye tracking and the implied scan out latency for individual subsets of pixels. Using this approach, little-to-no re-projection adjustments are applied to the pixel values at the location of the display(s) where the user is looking during image presentation, which mitigates unwanted, re-projection-based visual artifacts in a “region of interest” where the user is looking. In general, when large-scale re-projection adjustments are applied to an application-rendered frame, the resulting re-projected frame will exhibit unwanted visual artifacts, such as judder, which manifests as a moving or animating object that splits apart into two or more objects, frame-to-frame. By contrast, when small-scale re-projection adjustments are applied to an application-rendered frame, the resulting re-projected frame will mitigate these unwanted visual artifacts caused by re-projection. Thus, with the goal of making as accurate of a prediction as possible to render the frame correctly, and to make as few re-projection adjustments as possible during re-projection, the techniques and systems described herein improve the image quality in the region of interest where the user is looking by virtue of minimizing the re-projection adjustments to the pixel values in the region of interest. This improves the display quality of a HMD that implements a rolling band display driving technique.

[0013] In an illustrative example, a HMD may be worn by a user for purposes of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment. One or more display panels of the HMD present images based on frames that are output by an application (e.g., a video game), and these images are viewed by a user through the optics that are included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment. The HMD may utilize a rolling band technique to drive a display panel(s) of the HMD by illuminating individual subsets of the pixels in sequence over an illumination time period. Thus, the illumination propagates (or “rolls”) across the display panel(s) during the presentation of an image for a given frame.

[0014] The HMD system, including the HMD, may also utilize a technique called “re-projection” to compensate for slight inaccuracies in an original pose prediction of the HMD and/or to compensate for the application failing to make frame rate, which has the same effect as an original pose prediction that is slightly inaccurate. For example, a re-projected frame can be generated using pixel data from an application-rendered frame by transforming (e.g., through rotation and re-projection calculations) the application-rendered frame in a way that accounts for an updated prediction of the pose of the HMD.

[0015] Described herein are techniques and systems for mitigating unwanted visual artifacts caused by re-projection in a region of interest on the display(s) where a user is looking, thereby improving the display performance by making these unwanted visual artifacts, if present, unnoticeable to the viewing user because they will not manifest in the region of interest. In an example process, before providing pose data as input to an application for rendering the next frame, logic of the HMD system (e.g., a compositor) may predict a location on the display panel where a user (who is wearing the HMD) will be looking during the illumination time period for the frame. This prediction may be based at least in part on eye tracking data generated by an eye tracking system of the HMD system. In some embodiments, the logic of the HMD system is configured to predict a gaze point where the user will be looking during the illumination time period based on the eye tracking data, and to determine a location on the display panel that corresponds to the predicted gaze point. The logic of the HMD system is also configured to determine a time, within the illumination time period, at which an individual subset of the pixels–that corresponds to the predicted location on the display panel–will be illuminated. Using head tracking data generated by a head tracking system of the HMD system, the logic can then predict a pose that the HMD will be in at the determined time (which was dynamically targeted based on the eye tracking data indicating where the user will be looking during the illumination time period). Pose data indicative of the predicted pose of the HMD can then be sent to the application for rendering a frame, and logic of the HMD system, upon receiving the frame from the application, can apply re-projection adjustments to the pixel data for the application-rendered frame to obtain modified pixel data associated with a re-projected frame. This modified pixel data can be output to a frame buffer for presenting an image on the display panel(s), whereby the individual subsets of the pixels are illuminated in sequence over the illumination time period to present the image.

[0016] By dynamically targeting the render time for an application-rendered frame based on eye tracking and the implied scan out latency for individual subsets of pixels, the scene (or picture) can be rendered for a time at which light (photons) from the region of interest on the display panel(s) will actually reach the user’s eye. This means that the scene can be rendered correctly in the region of interest, and, hence, merely small-scale re-projection adjustments, if any, are applied to the pixel data (values) in the region of interest where the user is looking during image presentation. In other words, if, by tracking the user’s eyes, it can be determined that the user is likely to look at subset of the pixels within the top portion of the display panel(s) during the rolling illumination time period, logic (e.g., the compositor) of the HMD system can tell the application to render the scene in accordance with the user’s head pose that is predicted for the time when that subset of the pixels will illuminate during the rolling illumination time period. If, for a subsequent frame, the user is predicted to look at a subset of the pixels within the bottom portion of the display panel(s) during the rolling illumination time period, the logic can tell the application to render the scene in accordance with the user’s head pose that is predicted for the time when that subset of the pixels will illuminate during the rolling illumination time period. Thus, instead of always rendering the given frame for a time when the middle row of pixels will illuminate, the techniques described herein are directed to dynamically selecting a render time for the next frame based on where on the display panel(s) the user is predicted to look during image presentation. Said another way, for each frame, the logic of the HMD system is configured to tell the application which point in time to choose for the “picture of the world” that is to be presented on the display panel(s), and this point in time is chosen based on a prediction of where the user will be looking when that picture of the world is presented.

[0017] Also disclosed herein are systems, for example, systems including a display system (e.g., a HMD system), configured to implement the techniques and processes disclosed herein, as well as non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications, including, without limitation, non-VR applications (e.g., AR applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.

[0018] FIG. 1 is a diagram illustrating an example technique for dynamically targeting a render time for a given frame based on eye tracking. FIG. 1 depicts a head-mounted display (HMD) 100 worn by a user 102. The HMD 100 in the example of FIG. 1 may include a single display panel 104 or multiple display panels 104, such as a left display panel and a right display panel of a stereo pair of display panels. The one or more display panels 104 of the HMD 100 may be used to present a series of image frames (herein referred to as “frames”) that are viewable by a user 102 wearing the HMD 100. It is to be appreciated that the HMD 100 may include any number of display panels 104 (e.g., more than two display panels, a pair of display panels, or a single display panel). Hence, the terms “display panel,” as used in the singular herein, may refer to either display panel 104 of a pair of display panels of a two-panel HMD 100, or it may refer to a single display panel 104 of a HMD 100 with any number of display panels (e.g., a single-panel HMD 100 or a multi-panel HMD 100). In a two-panel HMD 100, a stereo frame buffer may render, for instance, 2160.times.1200 pixels on both display panels of the HMD 100 (e.g., 1080.times.1200 pixels per display panel).

[0019] The display panel(s) 104 of the HMD 100 may utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements (e.g., light emitting diodes (LEDs)) to emit light during presentation of frames on the display panel(s) 104. As an example, display panels 104 of the HMD 100 may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display technology for HMD applications.

[0020] The display panel(s) 104 of the HMD 100 may operate at any suitable refresh rate, such as a 90 Hertz (Hz) refresh rate. The “refresh rate” of a display is the number of times per second the display can redraw the screen. The number of frames displayed per second may be limited by the refresh rate of the display. Thus, a series of frames may be processed (e.g., rendered) and displayed as images on the display such that a single frame of the series of frames is displayed with every screen refresh. That is, in order to present a series of images on the display panel(s) 104, the display panel(s) 104 may transition from frame-to-frame, in the series of frames, at the refresh rate of the display, illuminating the pixels at every screen refresh. In some embodiments, the frame rate can be throttled, and phantom frames (based on re-projection) can be inserted between application-rendered frames.

[0021] The display system of the HMD 100 may also implement a rolling band type of display driving scheme. That is, the light emitting elements of the display panel(s) 104 may be individually-addressable such that individual subsets of the light emitting elements, and, hence, individual subsets of the pixels, can be illuminated independently and sequentially in a rolling band of illumination during an illumination time period. In some embodiments, both the array of pixels and the light emitting elements on the display panel(s) 104 are arranged in rows and columns, but not necessarily with a one-pixel per one-light emitting element correspondence. In this configuration, individual rows and/or individual columns of light emitting elements may be addressed in sequence, and/or individual groups of contiguous rows and/or individual groups of contiguous columns of light emitting elements may be addressed in sequence. As a consequence of addressing the light emitting elements in this “rolling” manner, the subsets of pixels that correspond to these individually-addressable subsets of light emitting elements may be “illuminated” independently.

[0022] As used herein, “illuminating a pixel” means illuminating the light emitting element that corresponds to that pixel. For example, a LCD illuminates a light emitting element of a backlight to illuminate the corresponding pixel(s) of the display. Furthermore, as used herein, a “subset of pixels” may comprise an individual pixel or multiple pixels (e.g., a group of pixels). In some embodiments, a subset of pixels includes a row of pixels, a column of pixels, a group of contiguous rows of pixels, or a group of contiguous columns of pixels. Thus, in an aspect of the techniques and systems described herein, subsets of pixels can be scanned out and illuminated in sequence (sequentially), such as by scanning out and illuminating each row of pixels in sequence, starting with a first row of the pixels (e.g., a top row of pixels) and ending with a last row of the pixels (e.g., a bottom row of pixels). However, any suitable pattern of illumination can be employed using the techniques and systems described herein (e.g., a snake-like pattern of illumination, column-by-column illumination, multiple rows/columns of pixels at a time in sequence, etc.).

[0023] In order to drive the display panel(s) 104 in this “rolling” manner, the HMD 100 may include, among other things, a display controller(s), display driver circuitry, and similar electronics for driving the display panel(s) 104. Display driver circuitry may be coupled to an array of light emitting elements of the display panel(s) 104 via conductive paths, such as metal traces, on a flexible printed circuit. In an example, a display controller(s) may be communicatively coupled to the display driver circuitry and configured to provide signals, information, and/or data to the display driver circuitry. The signals, information, and/or data received by the display driver circuitry may cause the display driver circuitry to illuminate the light emitting elements in a particular way. That is, the display controller(s) may determine which light emitting element(s) is/are to be illuminated, when the element(s) is/are to illuminate, and the level of light output that is to be emitted by the light emitting element(s), and may communicate the appropriate signals, information, and/or data to the display driver circuitry in order to accomplish that objective.

[0024] Pixel data for a given frame can be output to a frame buffer for presenting the frame as an image on the display panel 104. Pixel data for each frame may, in some embodiments, include a two-dimensional array of per-pixel values (e.g., color values). In some embodiments, the pixel data further includes additional data or metadata, such as depth values. In some embodiments, pixel data may include data for each pixel that is represented by a single set of color and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels). This pixel data can be output to a frame buffer (e.g., a stereo frame buffer) to present on image(s) on the display panel(s) 104 of the HMD 100 with a desired visual effect.

[0025] The HMD 100 may represent a VR headset for use in VR systems, such as for use with a VR gaming system. However, the HMD 100 may additionally, or alternatively, be implemented as an AR headset for use in AR applications. In AR, a user 102 sees virtual objects overlaid on a real-world environment, whereas, in VR, the user 102 does not see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panel(s) 104 and the optics (e.g., lenses) of the HMD 100. Examples described herein pertain primarily to a VR-based HMD 100, but it is to be appreciated that the HMD 100 is not limited to implementation in VR applications.

[0026] In general, a graphics-based application 106 (e.g., a video game) executing on a computing device–such as the HMD 100 itself, or a computing device (e.g., a personal computer (PC), game console, etc.) associated with, and coupled to, the HMD 100 as part of a HMD system–may be configured to output a series of frames 108(1), 108(2), 108(3), and so on (collectively 108). The series of frames 108 are ultimately presented as images on the display panel(s) 104 of the HMD 100. The example of FIG. 1 depicts three example frames 108(1) (or frame “F”), 108(2) (or frame “F+1”), and 108(3) (or frame “F+2”) with respect to a rendering timeline 110 to illustrate how the frames 108 can be rendered in series. Here, the application 106 renders frame F first, then frame F+1, and then frame F+2, in sequence, from left to right on the rendering timeline 110. The rendering timeline 110 also shows the rendering workloads 112 of a compositor 114 of the HMD 100 (or HMD system) towards the end of each rendering interval for each frame 108. An individual rendering workload 112 of the compositor 114 for a given frame 108 may represent adjustments that are applied to the pixel data output by the application 106 before rendering a final image on the HMD 100. Such adjustments may include, without limitation, adjustments for chromatic distortion, panel masking, re-projection, and the like, which are applied to the frame 108 output by the application 106 before rendering a final image on the HMD 100. Accordingly, the frames 108 that are shown in FIG. 1 are meant to represent “actual” frames in the sense that they are output from the application 106, which may represent a video game application, or any other type of graphics-based application. The application 106 may be executed in a graphics pipeline that outputs pixel data, and the compositor 114 is configured to modify that pixel data, and to output the modified pixel data to a frame buffer (e.g., a stereo frame buffer).

[0027] During runtime, an eye tracking system of the HMD 100 (or HMD system) may generate eye tracking data about a gaze point where the user 102 is looking at any given moment in time. The eye tracking system may include a camera or other optical sensor inside the HMD 100 to capture image data of a user’s eyes 116, and the captured image data can be used to determine motion vectors, interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye 116 relative to HMD 100, including a magnitude of torsion and rotation (e.g., roll, pitch, and yaw) and gaze directions for each eye 116. In one example, infrared light is emitted within the HMD 100 and reflected from each eye 116. The reflected light is received or detected by a camera of the HMD 100 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye 116. In some embodiments, the eye tracking system may integrate information from past measurements, measurements identifying a position of a user’s 102 head, and 3D information describing a scene presented on the display panel(s) 104. Thus, information for the position and orientation of the user’s 102 eyes 116 can be used to determine the gaze point in a virtual scene presented by HMD 100 where the user 102 is looking. Many methods for tracking the eyes 116 of the user 102 can be employed by the eye tracking system of the HMD 100 (or HMD system), and these are merely examples of eye tracking techniques that can be employed.

[0028] In any case, the eye tracking data generated by the eye tracking system of the HMD 100 (or HMD system) can be used by the compositor 114 to predict a gaze point where the user 102 will be looking during an illumination time period for the upcoming frame 108, and this gaze point can be correlated to a location on the display panel(s) 104, which can be defined at any suitable level of granularity (e.g., an individual subset (e.g., row) of the pixels, an individual group of contiguous subsets (e.g., rows) of the pixels, other regions of the display panel(s) 104, etc.). In some embodiments, the most recent eye tracking data (e.g., the gaze point) can be used as a proxy for predicting where the user 102 will be looking during the illumination time period for an upcoming frame 108. With today’s eye tracking technology, it is difficult to accurately predict where the user’s eyes will be at a future time because of the ballistic motions that can be exhibited by the eyes during rapid eye movement. This is one reason for using the most recent gaze point estimation as a proxy for predicting a future gaze point. However, as eye tracking improves over time, the prediction of where the user 102 will be looking at a future time may become more accurate, and, hence, a prediction of a future gaze point may be determined as a function of past eye tracking data. For instance, a future gaze point can be predicted based on motion vector estimations of the user’s eyes 116, and/or based on additional data.

[0029] FIG. 1 also shows a “scan out+illumination” timeline 118 to illustrate the illumination time periods corresponding to each frame 108 that is rendered in the rendering timeline 110. For example, after frame F is rendered by the application 106, and after the compositor 114 performs its rendering workload 112(1) to modify the pixel data for frame F, an image corresponding to frame F is presented on the display panel(s) 104 during the illumination time period 120(1) for frame F. During this illumination time period 120(1), subsets of pixel values (pixel data) for frame F are scanned out to the display panel(s) 104 via a display port (e.g., a high-definition multimedia interface (HDMI)), in sequence, and the light emitting elements of the display panel(s) 104 are illuminated in sequence to cause individual subsets of the pixels to illuminate in sequence. FIG. 1 illustrates an example where the individual subsets of pixels correspond to individual rows of pixels. For example, the display panel(s) 104 may have multiple rows of pixels, starting with a first row (“Row 1”), which may be the top row, and ending with a last row (“Row N”), which may be the bottom row. The illumination time period 120 may also be described in terms of the times t.sub.1 to t.sub.N shown in FIG. 1. For example, the first subset of pixels (e.g., “Row 1”) may illuminate at time, t.sub.1, and the last subset of pixels (e.g., “Row N”) may illuminate at time, t.sub.N, while the intermediate subsets of pixels may illuminate at respective times between these outer bounds of the illumination time period 120. As mentioned, for a 90 Hz refresh rate, the illumination time period 120 may be roughly 11 ms in duration.

[0030] In the example of FIG. 1, in order to render frame F, the compositor 114 may predict the gaze point where the user 102 will be looking during the illumination time period 120(1) for frame F. In the example of FIG. 1, the predicted gaze point for frame F is a gaze point that corresponds to an individual subset (or a group of contiguous subsets) of the pixels within a top portion of the display panel(s) 104. This may be described as the compositor 114 predicting a location of the display panel(s) 104 where the user 102 will be looking during the illumination time period 120(1), and, in this case, the predicted location for frame F is a location that corresponds to an individual subset (or a group of contiguous subsets) of the pixels within the top portion of the display panel(s) 104. It is to be appreciated that a “location” of the display panel(s) 104 can be defined in any suitable manner and/or at any suitable level of granularity suitable for describing where, on the display panel(s) 104 the user 102 may be looking. The example of FIG. 1 illustrates an example where the location of the display panel(s) 104 can be determined based on the rows of pixels of the display panel(s). Depending on the accuracy of the eye tracking system of the HMD 100 (or HMD system), and the resolution at which the individual subsets of pixels can be addressed, the prediction of where the user 102 will be looking can be made at any suitable level of granularity, such as by estimating that the user 102 is looking at a given subset(s) of pixels (e.g., an individual row(s) of pixels, an individual column(s) of pixels, etc.), or individual groups of contiguous subsets (e.g., groups of contiguous rows, groups of contiguous columns, etc.) of pixels.

[0031] Once the compositor 114 determines where the user 102 will be looking when the image corresponding to frame F is presented, the compositor 114 may be further configured to determine a time, within the illumination time period 120(1) for frame F, at which an individual subset of the pixels that corresponds to the location on the display panel(s) 104 (where the user 102 is predicted to be looking) will be illuminated. For example, if the user 102 is predicted to be looking at an individual subset (or a group of contiguous subsets) of the pixels within the top portion of the display panel(s) 104 during the illumination time period 120(1) for frame F, the compositor 114 may determine a time at which that subset of the pixels (within the top portion of the display panel(s) 104) will be illuminated. In an example, the compositor 114 may determine that the first row of pixels (e.g., “Row 1”)–which is within the top portion of the display panel(s) 104–will illuminate at time t.sub.1. In this example, the compositor 114 may determine a target render time (within the illumination time period 120(1) for frame F) as target render time=t.sub.1, because the user 102 is predicted to be looking at a location corresponding to the first row of pixels within the top portion of the display panel(s) 104 during the illumination time period 120(1) for frame F, based on the eye tracking data. As another example, if the user 102 is predicted to look at the 200.sup.th row of pixels, for example, the compositor 114 may determine, as the target render time, an illumination time of the 200.sup.th row of pixels (.

[0032] In addition to the eye tracking system, a head tracking system of the HMD 100 (or HMD system) may generate head tracking data about the pose of the HMD 100, and the compositor 114 is configured to predict a pose that the HMD 100 will be in at the determined time (the targeted render time). Thus, for frame F, the compositor 114 may predict a pose that the HMD 100 will be in at the time, t.sub.1, which was determined to be the time when the subset of pixels (e.g., “Row 1”) within the top portion of the display panel(s) 104 will be illuminated during the illumination time period 120(1). The compositor 114 may send pose data indicative of the predicted pose to the application 106 for rendering frame F. Providing the pose data to the application 106 in advance allows the application 106 to output pixel data for rendering imagery on the HMD 100 in a way that is correct for the user’s 102 predicted head pose at the future, targeted render time, t.sub.1. This means that the application 106 renders a scene that is appropriate for the user’s predicted head pose at time, t.sub.1, when light from the portion of the display panel(s) 104 where the user 102 is predicted to be looking reaches the user’s eye(s) 116.

[0033] The graphics logic of the HMD 100 (or HMD system) may be asynchronous, or synchronous. In an asynchronous system, the compositor 114 runs separate (on a separate, asynchronous thread) from the application 106 on a graphics processing unit (GPU) of the HMD 100 (or HMD system). For instance, the application 106 may call a function to receive pose data from the compositor 114, and the compositor 114 may provide the application 106 with the requested pose data (predicted to the target render time, t.sub.1, for frame F) so that the application 106 can render the frame 108 (e.g., frame F) according to the pose data, which corresponds to a virtual camera pose used to render the scene. Assuming the application 106 finishes rendering the frame 108 before the compositor’s workload 112 starts, the compositor 114 is configured to take the frame 108 (e.g., left and right image frames) from the application 106 and distort the frame(s) 108 into the back buffer(s) onto the display panel(s) 104 of the HMD 100. For example, the compositor 114 may perform, without limitation, chromatic distortion, panel masking, re-projection, and the like, before rendering a final image on the HMD 100 based on the frame 108 output by the application 106.

[0034] The compositor 114 may execute in a high-priority context mode, which means that when it is time for the compositor 114 to start its workload 112, the GPU driver allows the compositor 114 to preempt the application 106 (e.g., by interrupting the application 106 from rendering, if it is still rendering a frame 108, and/or preventing the application 106 from starting to render a next frame 108). The compositor 114 may allocate a time slot (up to 1 ms on average) at the end of each rendering interval to do its work 112, regardless of what is happening with the application 106. Thus, at every rendering interval, the compositor 114 renders “something” in the sense that the compositor 114 obtains the best frame 108 it can obtain from the application 106 (e.g., either a fresh/new frame 108, or a previously-rendered frame 108), and the compositor 114 uses the pixel data associated with that frame 108 to put pixel data in the frame buffer for output on the display panel(s) 104 of the HMD 100. The compositor’s 114 ability to output different pixel data for each screen refresh, no matter what, is the mechanism that keeps everything “live” and keeps the images rendered on the HMD 100 from hitching badly when the application 106 is not making frame rate.

[0035] By dynamically targeting the render time as a time (e.g., t.sub.1 for frame F) at which a subset of pixels nearest to where the user will be looking will illuminate, the image corresponding to frame F can be presented during the illumination time period 120 with merely small-scale re-projection adjustments, if any, at a location on the display panel 104 where the user 102 is looking during the illumination time period 120. This means that the user 102 will not see unwanted visual artifacts caused by re-projection with respect to moving or animating objects in the region of interest where the user is looking during the presentation of the image.

[0036] After frame F is rendered, the dynamic render time targeting may iterate for each frame in the series of frames 108, as they are rendered. For example, for frame F+1, the eye tracking data may indicate that the user 102 will be looking at an individual subset (or a group of contiguous subsets) of the pixels within the middle portion of the display panel(s) 104 during the illumination time period 120(2) for frame F+1, which causes the compositor 114 to dynamically target, as the render time for frame F+1, a time at which that subset (or group of subsets) of pixels in the middle portion of the display panel(s) 104 will illuminate. For example, the compositor 114 may determine that the middle row of pixels will illuminate at time,

t N 2 , ##EQU00001##

and, thus, may predict a pose that the HMD 100 will be in at time,
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

SolKutTeR · 31. Juli 2020

Neues und interessantes Valve Patent eines Wireless Konzept ähnlich dem Wireless VR-Kit für HTC HMD's.

Wireless Data Transport System for Head-Mounted Displays
Systems and methods for providing low latency and high bandwidth wireless data transport for various applications, such as virtual reality, augmented reality, or video applications. A wireless data transport system is provided that includes an electrically steerable antenna, such as a phased-array antenna, that is operative to selectively steer its beam based on control input. The wireless data transport system includes a tracking subsystem that is operative to track a mobile wireless device (e.g., head-mounted display (HMD), tablet computer, smart phone) as the mobile wireless device moves around in a tracked volume. The wireless data transport system utilizes the known position, orientation, or movement of the mobile wireless device receiving the data (e.g., video data) from the tracking subsystem and compensates for movement of the mobile wireless device by selectively adjusting the beamforming pattern of the steerable antenna based at least in part on the tracking data received from the tracking subsystem. Weiterlesen →

SolKutTeR · 5. Februar 2021

Recht interessante Kooperation mit Tobii und OpenBCI (Open Source Brain-Computer Interfaces):

Tobii, Valve, and OpenBCI Engaging in Research Collaboration to Make VR Gaming More Immersive
Tobii, the world leader in eye tracking, today confirmed that it is engaging in research collaboration with Valve and OpenBCI by incorporating Tobii’s eye tracking technology with elements of Valve’s Index hardware to produce developer units for the recently announced Galea Beta Program.

“We are excited to work with Valve and OpenBCI to explore the future of immersive gaming by combining the power of Tobii eye tracking and OpenBCI’s advanced brain computer interface technology,” said Anand Srivatsa, Division CEO of Tobii Tech.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

OpenBCI is an advanced technology company providing an open-source platform for applications related to brain computer interfacing (BCI). Valve, one of the most influential companies in the gaming industry, has long been a pioneering force for the evolution of immersive gaming, entertainment content delivery, and the advancement of virtual reality hardware and experiences.

Galea is a hardware and software platform that merges next-generation biometrics with mixed reality. It is the first device that integrates EEG, EMG, EOG, EDA, PPG, and image based eye-tracking into a single headset. The Galea Beta units will be built with elements from the Valve Index VR hardware.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

Developers interested in using the OpenBCI Galea headset to explore how neurotechnology can bring about the next generation of immersive digital experiences are invited to visit OpenBCI’s beta project website. Initial developer units are expected to ship in limited quantities to beta program participants in early 2022.

The VR market is poised to grow into several tens of million units in volumes over the next three to five years, which represents an attractive opportunity for Tobii.

Tobii does not consider this collaboration as an official design win at this stage.
Klicke in dieses Feld, um es in vollständiger Größe anzuzeigen.

SolKutTeR · 18. März 2021

Weiteres Valve Patent welches auf ein Wirelesskonzept beruht.

Adjustable Head-Mounted Display
A head-mounted display including a front, a back, a first actuator, a second actuator, a first adjustable member extending between the front and the back and operably engaging the first actuator, and a second adjustable member extending between the front and the back and operably engaging the second actuator. The first adjustable member and the second adjustable member may be adjustable in length via the second actuator to vary a gap distance between the front and the back. In some embodiments, the head-mounted display may include a wire routing assembly for routing wire(s) between the front and the back of the head-mounted display. Additionally, in some embodiments, the head-mounted display may include a harness for engaging a user. Weiterlesen →

Valve und die zukünftigen Pläne mit VR

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

Nützliche Suchen

Valve und die zukünftigen Pläne mit VR

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

SolKutTeR ADMIN VRF Team

Nützliche Suchen

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team

SolKutTeR

ADMIN
VRF Team