Deep Learning

DeepFaceLab — DeepFaceLab is a tool that utilizes machine l...

🍂 Who doesn’t dream of a new FPGA family that can provide embedded hard neurons in its silicon architecture fabric instead of the conventional DSP and multiplier blocks? The optimized hard neuron design will allow all the software and hardware designers to create or test different deep learning network architectures, especially the convolutional neural networks (CNN), more easily and faster in comparing to any previous FPGA family in the market nowadays. The revolutionary idea about this project is to open the gate of creativity for a precise-tailored new generation of FPGA families that can solve the problems of wasting logic resources and/or unneeded buses width as in the conventional DSP blocks nowadays. The project focusing on the anchor point of the any deep learning architecture, which is to design an optimized high-speed neuron block which should replace the conventional DSP blocks to avoid the drawbacks that designers face while trying to fit the CNN architecture design to it. The design of the proposed neuron also takes the parallelism operation concept as it’s primary keystone, beside the minimization of logic elements usage to construct the proposed neuron cell. The targeted neuron design resource usage is not to exceeds 500 ALM and the expected maximum operating frequency of 834.03 MHz for each neuron. In this project, ultra-fast, adaptive, and parallel modules are designed as soft blocks using VHDL code such as parallel Multipliers-Accumulators (MACs), RELU activation function that will contribute to open a new horizon for all the FPGA designers to build their own Convolutional Neural Networks (CNN). We couldn’t stop imagining INTEL ALTERA to lead the market by converting the proposed designed CNN block and to be a part of their new FPGA architecture fabrics in a separated new Logic Family so soon. The users of such proposed CNN blocks will be amazed from the high-speed operation per seconds that it can provide to them while they are trying to design their own CNN architectures. For instance, and according to the first coding trial, the initial speed of just one MAC unit can reach 3.5 Giga Operations per Second (GOPS) and has the ability to multiply up to 4 different inputs beside a common weight value, which will lead to a revolution in the FPGA capabilities for adopting the era of deep learning algorithms especially if we take in our consideration that also the blocks can work in parallel mode which can lead to increasing the data throughput of the proposed project to about 16 Tera Operations per Second (TOPS). Finally, we believe that this proposed CNN block for FPGA is just the first step that will leave no areas for competitions with the conventional CPUs and GPUs due to the massive speed that it can provide and its flexible scalability that it can be achieved from the parallelism concept of operation of such FPGA-based CNN blocks.
🍂 This pet project is in recognition of the upcoming 40th Anniversary of Star Trek the Motion Picture (TMP), which I thought could also be a geeky experiment to test the evolving deep learning technology.  One of the criticisms often levelled against TMP is that it is heavy on special effects and lacks some of the camaraderie of the original series (TOS).  I wondered if there were any little tweaks that could be done to certain scenes using Deepfake or Deep Video Portrait technology to nudge the mood slightly. I settled my focus on scenes involving the actress Grace Lee Whitney [].  She is a personal favourite of mine, having been the female lead in season one of TOS who was fired, allegedly after being sexually assaulted by an executive, and who was asked to reprise her role in the franchise for TMP.  Her appearance is little more than a cameo with a couple of lines of dialogue, only one of which is delivered to the camera.  However, she also gets a cameo in Star Trek III which carried quite a bit of emotional resonance, and most of her appearance is in close up and therefore potentially useful for Deep Portrait technology.  So this is a proposal to test the possibility of Star Trek the Motion Picture: the Rand Cut. ********** Rand's final scene in TMP is where Dr McCoy beams aboard, [].  This scene is normally shortened in most edits because some of the dialogue makes light of an earlier disaster in which two crewmen die horribly. The flippant tone and inane grins are inappropriate considering the awful tragedy that has just taken place.  I wondered if the full scene could benefit from Deepfake / Deep Video Portrait to make any / all of the following changes: 1. Make Kirk and Rand look more serious until after McCoy arrives: A yeoman beams aboard and tells Kirk that McCoy, "Insisted we go first to see how it scrambled our molecules," which leads Kirk and Rand to smile.  If the smirking is replaced with serious faces and Kirk delivers his "That has a familiar ring to it," line with a straight face, it adds both a double meaning to his comment (it acknowledges the seriousness of the actual disaster at the same time as McCoy's fan-favoured reluctance to trust the transporter) as well as being consistent with Kirk's long-established gallows humour. NB: The earlier scene where the disaster takes place has lengthy close ups of Kirk and Rand looking very serious, which could work for Deep Video Portrait adjustment [] 2. Insert a line of dialogue for Janice Rand: In the fan-created episode, World Enough and Time (WEAT) [], Rand says to Sulu, "Don't look so worried."  Inserting this line into TMP after the Yeoman speaks but before Kirk's line could also work on multiple levels if delivered with a more serious face (reassuring Kirk that the problem that caused the earlier disaster has now been fixed, a show of empathy alluding to Rand's previous close working relationship with Kirk, and an acknowledgement that Kirk is nervous about his reunion with McCoy).  NB: WEAT was filmed many years after TMP when Grace was in her seventies so it seems that Deep Video Portrait might be needed to help make the younger actress's mouth move appropriately in the TMP footage? NB: Although the line is brief, it's possible that there is not quite enough of a pause in between the yeoman's and Kirk's lines to fit in the extra dialogue.  Another possibility is to insert it off camera, just after McCoy arrives (I don't think it works quite as well there as it feels like she should say, "Don't look so worried, Dr McCoy," which would require extra splicing from TOS dialogue from either Charlie X or the Corbomite Manoeuvre (where she mentions McCoy by name) and splicing her 70 year old voice with her 36 year old voice might sound odd, albeit still possible with more sound editing effort and less Deep Video Portrait effort).  A third option is to insert it into the brief shot where she is smiling after McCoy's arrival just before Kirk steps out of the booth (my concern here is that, again, it seems to make light of the earlier disaster). Grace only appears in the first third of TMP and it seems a shame that she vanishes so early on after such a long absence from the franchise due to an indemic problem only recently given public recognition by the Me Too movement. The second, more ambitious, fantasy edit would be to digitally add the actress into some later scenes. One possibility is to insert a couple of close-ups into the very long V'Ger flyover.  Rand is not established to be on the bridge at this point but there are a lot of open-mouthed stares from other characters in this scene so I'm sure another few from Rand would not be out of place. The main problem here is finding a suitable clear background shot on which to paste her image. Another option is to insert a brief rear shot from the Rec Deck scene into a scene where Kirk is being dressed in a space suit to pursue Spock []. The problem here I think is that the scene might be a bit too drawn out to use the brief rear shot from the earlier scene effectively. Not wanting to be overly ambitious, I picked a scene in the latter part of the movie where Kirk and McCoy are watching Decker and the Ilia probe on a viewscreen in Kirk's quarters.  Earlier in the movie, McCoy chews out Kirk and leaves these same quarters.  As McCoy heads out the door, he becomes small enough to be obscured if a close up of Rand were to be digitally added in the foreground. So for this more complex edit I wondered if it would be possible to do the following : 1. Paste a close up Rand into a static image of Kirk's quarters. After the establishing shot of Kirk and McCoy watching the viewscreen, cut to the new view of the quarters, angled facing the door, taken from the earlier scene. Paste a digital copy of Rand in the foreground (to obscure McCoy's original exit) and establish her presence in the room. 2. Insert a line of dialogue for Janice Rand. Use Deep Video Portrait to allow her to speak a line of dialogue from the TOS episode the Balance of Terror, "Can I get you something from the galley, sir?" It's trivial but she used to be his yeoman with knowledge that he forgets to eat in a crisis (the Corbomite Manoeuvre), and she also has a history of spending her personal time delivering food to her friends who are working overtime (the Man Trap). The original line in Balance of Terror was delivered in close up so it might be fairly easy to port across onto an older TMP version of the actress. 3. Insert a line of dialogue for Kirk. Insert an off camera, "No thank you," "Thank you," or some such from Kirk.  This would allow the focus to stay on the close up of Rand to minimise any complicated cuts to the original footage. If it's possible for her to show facial recognition of his reply (pursed lips or a slight nod perhaps) it might add to the realism of the exchange. 4. Insert a line of dialogue for Janice Rand. After a slight pause, use Deep Video Portrait to allow her to speak a line of dialogue from the TOS episode Miri, "Do you suppose she knows?" 5. Insert a line of dialogue for Dr McCoy. Cut back to the original footage but add the Dr McCoy line from the TOS episode, Spock's Brain, "Jim, she may not remember, or even really know," and then let the remainder of the scene take place as normal. 6. Add a reaction shot of Rand. Although not strictly needed, you could add a second, brief reaction shot of Rand after Kirk thumps the table to show her own shock or disappointment when Decker's line of questioning (viewed by them on the screen) fails. As a final edit I would love Rand to be featured walking onto the bridge or shown as present on the bridge in the final scene of TMP.  This would mark the only moment, apart from the final scene in Star Trek IV when all nine original main characters are present in the same scene.  Unfortunately,  I don't think there is a suitable shot anywhere else in the movie that could work here. So there are some ideas of mine for 'the Rand Cut'.  Does anyone have any thoughts or suitable ideas for code that could make some or all of these scenes as viable for insertion into a movie Edit?